The world is in an arms race to create data centers in 2026, to run and train the most intelligent AI. We spent close to $1 trillion in Capex in 2026 for building data centers and GPUs. Recently free open weight LLMs / AI models have shrunk so you can host them on your laptops and phones. I decided to try these for coding tasks to see if they work.
My local AI setup
I did a little research and found that I could test local LLMs with a setup consisting of three parts: LM-Studio, OpenCode and a compact open weight model.
Hardware
I do not have a computer with an Nvidia graphics card for testing. I used my laptops.
- M2 MacBook with 16 GB RAM
- PC with AMD Ryzen AI 7 350 w/ Radeon 860M and 32 GB RAM
LM Studio
LM Studio is a program that can download a lot of free LLMs models. You can use it to chat with the LLM, but it can also serve LLMs up for coding.
It is not the most efficient tool for models serving, but it is simple and versatile.
OpenCode
OpenCode is an open source AI agent harness that runs in a terminal or as an app. It has momentum and 195k GitHub stars.
I liked its minimal cursor based GUI. It connects to LLMs, from frontier labs like Anthropic, Google and OpenAI, but also cheaper models and local models. This gives some protection against vendor lock-in, which can lead to rising AI cost.
OpenCode has good ergonomics, similar to Claude Code. But in OpenCode you always see the number of tokens and money spent. I don't know when the LLM is doing work for cheap and when it burns tokens by reading through long logs or keeping sending a long context to the LLM. When you always see the meter running you notice waste fast.
Compact open weight models
Gemma 4 E4B 6 GB
First I tested Gemma 4 E4B. It is a compact LLM around 6 GB, 8 billion parameters using 4 bit quantisation, developed by Google DeepMind. It could handle coding test tasks, without use of web searches. I tested it with C++, Makefile, Python, JavaScript and TypeScript.
Gemma gets confused when I ask for complicated work or to look at an existing code project. I used a context window of 22k tokens due to hardware limitation, maybe what part of the reason for its limitations.
It is running at a good speed. It feels like using CoPilot a couple of years ago. But when you are using it in the terminal it has a contemporary feel. Different variations of the Gemma 4 E4B model ran fine both on the MacBook and the PC.
Qwen3.5-9B 6 GB
Qwen3.5-9B from Alibaba is similar in size to Gemma 4 E4B. It could also finish simple coding tasks. Wrote examples in Python, JavaScript, TypeScript and Rust. Qwen did better on understanding existing projects, but still far from new frontier models.
Devstral Small 2 24B 9 GB
Slightly bigger code model with 24 billion parameters from Mistral.ai, a new French startup. It is aggressively compressed and I squeezed it onto my 32GB laptop. I gave it a Raylib code test. It handled the test in C, C++ and Rust. Also made graphics demos in JavaScript and ported it to TypeScript. Devstral quickly slowed down and also overheated my laptop.
Llama.cpp for more speed
Llama.cpp is an application for serving up LLMs. It runs faster and can handle bigger contexts than LM Studio. Using it feels smoother. But there is more work involved in using it, and the LLMs you downloaded with LM Studio will not necessarily work with Llama.cpp. On Linux I had to build it myself to get a new enough version.
Take away
All the LLMs of size 6 GB and up, run on consumer hardware and are good enough to do actual coding work.
IP over the wire and privacy
Sending private information or intellectual properties over the wire has many problems. Especially now that AI easily can extract and use information. AI panic makes companies take big risks in order not to be left behind. This has turned software development into a minefield.
This and regulatory requirements will probably mean that local AI will gain some market share.
Local AI is not fool proof. When Microsoft announced a new AI feature called Recall in Windows 11, with screen captures every few seconds, I stopped using Windows. There was big backlash against Recall, so Microsoft had to change Recall and make it strictly opt-in.
AI economics
I have no idea who will win the artificial intelligence race. Maybe the frontier lab can hold their lead due to better model training on bigger data centers. The hyperscalers could win on better cloud integration. The $5 or $20 AI subscriptions are only paying a fraction of AI investments.
Companies are now seeing an exploding AI bill. In a matter of months they have gone from leader boards for AI token spend, to employees having a limited monthly budget for tokens.
ChatGPT was synonymous with AI. Now AI is a competitive market place with big tech companies and AI startups competing on token cost. High cost has also contributed to increased interest in free open weight models.
For most tasks you do not need that much intelligence. I am not using the top frontier models for programming, cheaper AI models are enough.
Maybe the AI race is morphing into a race for super intelligence. Is super intelligence profitable, and profitable for what?
Use case for self hosting coding LLMs
Self hosted LLMs now have reached a level where they can do limited useful work. I ran 3 open weight models on my modest laptops and I was impressed. You have to be prepared to pair-program with your LLM. I would recommend self hosting LLMs if you are offline, broke or cannot send data or code over the internet for privacy or legal reasons. Your mileage may vary, but setup is easy.
A year ago AI = ChatGPT. Today AI is more like the Cambrian explosion with many evolutionary strategies.
