Self Hosting Coding LLMs

The world is in an arms race to create data centers in 2026, to run and train the most intelligent AI. We spent close to $1 trillion on building data centers. Recently free open weight LLMs / AI models has shrunk so you can host them on your laptops and phones. I decided to try these for coding tasks to see if they work.

My local AI setup

I did a little research and found that I could test local LLMs with a setup consisting of three parts: LM-Studio, OpenCode and a compact open weight model. 

Hardware

I do not have a computer with an Nvidia graphics card for testing. I used my laptops.

• M2 MacBook with 16 GB RAM
• PC with AMD Ryzen AI 7 350 w/ Radeon 860M and 32 GB RAM

LM Studio

LM Studio is a program that can download a lot of free LLMs models. You can use it to chat with the LLM, but it can also serve LLMs up for coding.

It is not the most efficient tool for models serving, but it is simple and versatile.

OpenCode

OpenCode is an open source AI agent harness that runs in a terminal or as an app. It has momentum and 195k GitHub stars.

I liked its minimal cursor based GUI. It connects to LLMs, from frontier labs like Anthropic, Google and OpenAI, but also cheaper models and local models. This gives some protection against vendor lock in, that can lead to rising AI cost.

OpenCode has good ergonomics, similar to Claude Code. But in OpenCode you always see the number of tokens and money spent. I don't know when the LLM is doing work for cheap and when it burns tokens by reading through long logs or keeping sending a long context to the LLM. When you always see the meter running you notice waste fast.

Gemma 4 E4B open weight model from Google

Gemma 4 E4B is a compact LLM with 4 billion parameters using 4 bit quantisation, developed by Google DeepMind. My first impression was that it could handle coding test tasks, without use of web searches. I tested it with C++, Makefile and Python.

Gemma gets confused when I ask for more complicated work. I used a context window of 22k tokens due to hardware limitation, maybe what part of the reason for its limitations.

It is running at a good speed. It feels like using CoPilot a couple of years ago. But when you are using it in the terminal it has a contemporary feel. Gemma 4 E4B model ran fine both on the MacBook and the PC. It comes in different variations, and is around 5GB.

IP over the wire and privacy

Sending private information or intellectual properties over the wire has many problems. Especially now that AI easily can extract and use information. AI panic makes companies take big risks in order not to be left behind. This has turned software development into a minefield.

This and regulatory requirements will probably mean that local AI will gain some market share. 

Local AI is not fool proof. When Microsoft announced a new AI feature called Recall in Windows 11, with screen captures every few seconds, I stopped using Windows. There was big backlash against Recall, so Microsoft had to change Recall and make it strictly opt-in.

AI economics

I have no idea who will win the artificial intelligence race. Maybe the frontier lab can hold their lead due to better model training on bigger data centers. The hyperscalers could win on better cloud integration. The $5 or $20 AI subscriptions are only paying a fraction of AI investments.

Companies are now seeing an exploding AI bill. In a matter of months they have gone from leader boards for AI token spend, to employees having a limited monthly budget for tokens.

ChatGPT was synonymous with AI. Now AI is a competitive market place with big tech companies and AI startups competing on token cost. High cost has also contributed to increased interest in free open weight models.

For most tasks you do not need that much intelligence. I am not using the top frontier models for programming, cheaper AI models are enough.

Maybe the AI race is morphing into a race for super intelligence. Is super intelligence profitable, and profitable for what?

Is self hosting coding LLMs viable?

My first impression is that self hosting LLMs now have reached a level where they can do limited useful work. I was able to run coding LLMs tests on underpowered laptops. Your mileage may vary, but setup is easy.

Intrigued? Then take it for a spin.