Local AI - Measuring Results - Part 1

I’ve been reading a lot about local LLMs. People seem pretty excited about it. I decided to give it a try.

A Reflection on LLMs

If you’re into AI, you probably know what an LLM is. In a very rough explanation, it’s an algorithm that produces output based on a prompt. It processes its response according to a huge dataset it was trained upon. The capabilities of an LLM generally depend on its training data.

Some popular ones are ChatGPT, Claude, and Gemini. In general, you interact with these LLMs through an application or website developed by its vendor.

But accessing the LLM through the vendor website is just one way of accessing it. You can use any other chat interface (often called harness) that supports LLM-compatible APIs and “talk” to LLMs in the same way. You can also in a similar fashion give tools to the LLM for agentic workflows. The only difference is that you don’t have the UX of the vendor. You’re getting the raw response from the LLM with all its capabilities.

Connecting to an LLM through an API or through any sort of interface assumes that it is running somewhere on the vendor’s datacenters. So whatever you type is going to their servers and being processed by the LLM. This is in general fine, but we’ve seen the price per tokens sky-rocket lately. Also, companies like OpenAI and Meta are not exactly what I would call trustworthy when storing my personal data. And let’s admit, some of us had personal conversations with LLMs at some point.

Other than price and privacy, some tasks might be too simple for LLMs with trillions of parameters. To avoid giving a dumb example like “how many Rs there is in strawberry”, let’s take a simple but common example for developers: explore a code base and explain its structure, or find a certain spot on the code where a specific logic happens. These cases could use very small LLMs, potentially local ones.

What Are Local LLMs?

The same way an LLM is running on a server somewhere, it can also run on your machine with certain limitations.

An LLM with hundreds of billions of parameters requires a lot of GPU computation. More than a regular person can ever purchase in their entire life. Check the news for the billions of USD spent on this to have an idea.

For a regular human being who doesn’t have access to billions of dollars, the solution is to downscale LLMs to the point they start fitting common hardware. By common hardware I mean some powerful domestic GPUs, like RTX 5090. Of course, this hardware is expensive, specially considering the spike on memory prices due to AI infrastructure demands. Still affordable for domestic users though.

Also, Mac CPUs after Apple Silicon share memory with the GPU. So if you have 48Gb of RAM, you can run some pretty decent models locally. The results from my research were made on Mac GPUs (aka Metal) on two different machines with 16Gb and 48Gb of RAM.

But running LLMs locally comes with a limitation: you can only run models up to a certain scale, and in general, the smaller the LLM, the less capable and smart it is.

However, depending on the task, a local LLM could do just fine. It’s just hard to measure. What parameter size is better? What vendor? Should I pick open source or closed source?

These are questions I want to start answering in this post.

How Do We Measure This?

One could say: “Well, you just need to check a benchmark and find an LLM that is smart enough and small enough for you, and there you go!”

The problem is that general benchmarks don’t predict task performance. If you test a model on MMLU or HumanEval, you learn something about the model, but nothing about whether it can actually solve your problems.

So, for this introductory post, I designed a more practical test: Can small local LLMs refactor real code?

The Test Design

I have a React component that renders a list of selectable items. It has a prop to enable sorting the items by selection. Selected items are first in case this prop is true.

I created two completely non-scientific, biased and fully opinionated tests based on it:

Test 1: Removal of the sorting prop

Remove the sorting prop from the React component and all underlying logic tied to it. Simple on paper. But it tests:

• Can the model understand scope? (only remove unused props)
• Can it track dependencies? (if a prop is used anywhere, it must stay)
• Can it simplify code without breaking it?

The objective is to have a component that renders the list as is regardless of selection with the cleanest code as possible.

Test 2: Inverse: adding the prop back

Starting from a React component without the prop and the sorting logic, add the same prop back with sorting logic. This tests:

• Can it implement a feature from requirements?
• Can it read instructions carefully? (sort by the selected field, not selectedItemId, for example)
• Does it produce clean, idiomatic code?

The objective is to have the same original component with sorting logic, as clean as possible.

The Test Setup

For each test, I measured five things:

• Correctness - Does it compile? Does it solve the problem?
• Speed - Tokens per second (how long does it take?)
• Quality - Is the code clean or full of unnecessary patterns?
• Tokens - How many tokens does it consume?
• Thinking - If the model supports thinking, how long does it take?

I’m using LM Studio to download and execute the models. Each execution starts from a completely pristine context with no tools, skills or MCPs enabled. Each model is given 8192 tokens of context. They all use exactly the same prompt.

The Contenders

I tested five models, all under 35B parameters:

• Phi-4 Mini 3.8B
• Mistral 7B
• GPT-oss 20B
• Qwen Coder 30B
• Gemma-4-e4b (9B)

What I Found

The results were surprising. Model size predicted almost nothing.

Key observations:

Deepseek Coder 33B, the largest and supposedly the most capable, was the slowest (9 tokens/sec) and produced buggy code. It was running on decent hardware, so the machine is not to blame.

Qwen 2.5 Coder 7B, at one-fifth the size, was 6x faster and correct.

Phi-4 hallucinated already on the thought process.

More interesting: Better prompts helped some models, not others. Clearer instructions improved code quality for small models but didn’t fix fundamental misunderstandings. Every model 20B and larger got the inverse test wrong in the same way. They confused “sort by the selected field” with “sort by the selected item ID.” This is a comprehension gap, not a prompt gap.

The Loop Experiment

Here’s where it got interesting. I’ve been reading about agentic loops lately. I decided to give this a try.

When I gave Qwen 3.5 9B explicit feedback about what went wrong—“You removed the wrong prop, keep keepLabel because it’s used in mapItemToEntry”—it fixed the bug on the second attempt.

Attempt 1 (minimal prompt): Failed. Removed the wrong prop from the interface.

Feedback injected: “Critical rule: Only remove props that are not used anywhere in the function body. If a prop appears in any line of code, it must stay in the interface.”

Attempt 2: Passed. Correctly kept keepLabel, only removed sortItems.

This changed everything. A model that fails alone but succeeds with feedback is exactly what you need for agentic work. Perhaps you don’t need a trillion parameters model that does right at first attempt.

What This Means

Three things stand out:

1. Small models work if you guide them.

Qwen 2.5 Coder 7B (7B) nailed both tests on the first try with the right prompt. Gemma-4-e4b (9B) produced production-quality code. You don’t need 30B+ parameters for refactoring tasks.

2. Model size is not the metric.

Deepseek 33B was the worst performer. Speed and correctness came from training quality and task alignment, not raw parameter count. The “biggest model wins” narrative is wrong.

3. Feedback loops matter.

Models fail in predictable ways. But they learn from explicit correction. A model that gets it wrong alone but right with feedback is not a failure, but the foundation of an agentic system.

But it’s not all fun and games

Of course, from “add a prop to this component” to the crazy agentic development world we see nowadays is a huge leap.

A small model performing well in such a simple task is not guarantee that it will be excellent for a heavier environment.

This opens up for future tests though. And I have some ideas in mind.

The Next Questions

I have a few open questions I will put to experiment in the next few days:

• If local models can refactor code with feedback, what else can they do?
• How do you build a system that uses cheap local iteration + smart verification to outperform expensive cloud APIs?
• Qwen 2.5 Coder 7B proved to be good with code. But it supports no tools. For an agentic system, could we combine multiple models taking advantage of what they are best at?

Maybe we don’t need Claude Fable 5 at maximum thinking to solve most of our problems after all.

We know Qwen 2.5 Coder 7b might be the sweet spot for coding. But I need to find the best one for tool use. Probably Gemma 4 e4b is a good start, since it performed well on this test.

Stay around to see the next experiments.

All tests were run on a 16GB or 48GB machine with LM Studio using Q4_K_M quantization. Full results and test methodology available on request.