How to Benchmark Local Coding Models Without Fooling Yourself

Once I had a local model running through Ollama, the obvious next question was: Is it actually good enough?

I could ask it a few Java questions, look at the answers, and decide whether they felt good. But if I wanted to compare multiple models later, "this one feels smarter" wouldn't tell me much.

It is surprisingly easy to make an informal benchmark unfair. Give one model a simple coding problem, another a more complicated one, help one when it makes a mistake, run one while the computer is idle, and suddenly the comparison doesn't mean much.

I wasn't trying to build a scientific LLM benchmark. I needed something much simpler: a repeatable set of tests that represented what I care about as a developer.

So before comparing models, I defined the benchmark.

What do I actually want to measure?

A coding model needs to do more than generate code.

Eventually, I want to put one of these models behind an agent that can work with an existing project. It will need to understand code, debug problems, follow project instructions, and make reasonable engineering decisions.

At the same time, I'm running everything locally. A brilliant model that consumes all 16 GB of memory and makes the rest of my computer painful to use isn't necessarily useful.

I separated the benchmark into two parts.

The first measures quality:

#	Test	What I'm looking for
1	Basic coding	Can it generate correct code?
2	Code understanding	Can it understand code it didn't write?
3	Debugging	Can it identify and fix a bug?
4	Reasoning	Can it evaluate engineering tradeoffs?
5	Architecture	Can it reason about interacting components?
6	Instruction following	Can it respect precise constraints?

The second measures experience:

#	Test
7	Cold-start response time
8	Warm response time
9	A more realistic coding response
10	Consistency across repeated prompts
11	Memory pressure and swap
12	Whether my Mac remains comfortable to use

This gives me two different questions:

Can the model do the work?

and:

Would I actually want to use it while working?

Test 1: Can it write basic code?

I started with a small Java problem:

Write a Java method that receives a List<Integer> and returns the first
duplicated value. Return Optional.empty() if there are no duplicates.
The solution should run in O(n) time.

This tests several things without being complicated.

The model needs to produce valid Java, understand Optional, choose an appropriate data structure, handle the no-duplicate case, and respect the O(n) requirement.

I'm looking for:

• correct syntax;
• correct behavior;
• an appropriate structure such as a HashSet;
• O(n) average time;
• no unnecessary complexity.

This isn't supposed to challenge an advanced coding model. It's the baseline. If a model struggles here, there isn't much reason to move directly into more complicated agentic work.

Test 2: Can it understand code?

Generating new code and understanding existing code are different tasks.

For the second test, I provide code rather than asking the model to create it:

public BigDecimal calculateTotal(List<OrderItem> items) {
    BigDecimal total = BigDecimal.ZERO;

    for (OrderItem item : items) {
        if (item.getQuantity() > 0) {
            total = total.add(
                item.getPrice()
                    .multiply(BigDecimal.valueOf(item.getQuantity()))
            );
        }
    }

    return total;
}

Then I ask:

Explain what this method does and identify any edge cases or assumptions
in its implementation.

Now I can see whether the model notices things such as a null list, null items, null prices, zero or negative quantities, and assumptions around monetary values.

I'm also interested in whether it invents problems that aren't supported by the code.

That's an important distinction. A long list of "possible edge cases" may look impressive while actually showing that the model isn't reasoning carefully about the types and information it received.

Test 3: Can it debug?

Next, I deliberately give it broken code:

public Map<String, Integer> countWords(List<String> words) {
    Map<String, Integer> counts = new HashMap<>();

    for (String word : words) {
        counts.put(word, counts.get(word) + 1);
    }

    return counts;
}

The prompt is:

This method throws an exception when processing a word for the first time.

Identify the problem, explain why it happens, and provide a corrected
implementation.

The important part isn't only whether it knows getOrDefault() or merge().

I'm looking for the complete reasoning:

Understand the code
        ↓
Identify the failure
        ↓
Explain the cause
        ↓
Produce a valid correction

Later, an agent will encounter compiler errors and failing tests. A model that is good only at generating code from a blank page won't be enough.

Test 4: Can it reason about tradeoffs?

This is where I start making the benchmark less comfortable.

You need to process 10 million transaction IDs and determine which IDs
appear more than once.

Memory is limited, and the transaction IDs are integers.

Describe two possible approaches, including their time and space complexity,
and explain when you would choose each approach.

There is no need to generate a large application here.

The interesting part is the phrase:

Memory is limited.

A model might immediately suggest putting everything into a HashSet. That's a perfectly reasonable duplicate-detection strategy, but does it stop and consider the memory constraint?

I want the model to discuss alternatives, their time and space complexity, and the conditions under which each makes sense.

This starts testing engineering judgment rather than syntax.

Test 5: Can it reason about architecture?

For this test, I chose something closer to the backend work I normally do:

Consider a Spring Boot REST API with this architecture:

Controller → Service → Repository → PostgreSQL

The application also uses Redis as a cache.

A GET /customers/{id} endpoint should:
1. Check Redis first.
2. If the customer is cached, return it.
3. Otherwise query PostgreSQL.
4. Store the result in Redis.
5. Return the customer.

Explain where this logic should live and why. Also describe what
responsibilities should remain in the Controller, Service, Repository,
and cache layer.

I'm intentionally not asking for code.

I want to know whether the model understands separation of concerns: where the orchestration belongs, what the Controller should be responsible for, what belongs in the Service, what the Repository should know about, and how Redis fits into the design.

This is important for the agent experiment because generating code that compiles isn't enough. An agent can create perfectly valid Java while putting the logic in completely the wrong place.

Test 6: Can it follow boring instructions?

The last quality test is technically easy:

Create a Java method that determines whether a string contains balanced
parentheses.

Requirements:
- Return boolean.
- Use only the Java standard library.
- Do not use regular expressions.
- Ignore all characters except '(' and ')'.
- Treat an empty string as balanced.
- Provide the solution first.
- Then provide exactly three short bullet points explaining it.

The balanced-parentheses problem isn't really what I'm testing.

I'm testing whether the model follows all the instructions.

Did it avoid regular expressions? Did it ignore other characters? Did it handle an empty string? Did it return the requested type? Did it provide exactly three bullets?

A model can produce technically correct code and still fail the task.

This becomes much more important once the instructions look like:

Modify only these files.
Preserve the existing public API.
Do not add dependencies.
Run the tests.
Do not commit.

Ignoring "exactly three bullets" is harmless. Ignoring "do not modify this file" may not be.

Don't reduce everything to "right" or "wrong"

I didn't want to record only whether each answer passed.

For every test, I evaluate three dimensions independently:

Dimension	Question
Correctness	Did it actually solve the problem?
Reasoning	Was the explanation technically sound?
Instructions	Did it follow what I asked?

I use a deliberately simple scale:

0 = Incorrect
1 = Partially correct
2 = Correct

I also record whether I had to intervene.

This isn't statistically sophisticated, and it doesn't need to be. The score exists mainly so I can run the same tests against another model and have something more concrete than memory.

The notes are often more valuable than the number.

Don't help the model during the test

This became one of the most important rules in the benchmark.

Suppose a model produces an incorrect answer.

My normal reaction would be:

That's not correct because you're ignoring the memory constraint.
Try again.

But if I do that, I'm changing what I'm measuring.

The result is now:

Model
+
My correction
+
Second attempt

rather than:

Model
+
Original prompt

So during these standalone tests, I record the first response and move on.

Later, when I test an actual agent, iteration will be part of the experiment. The agent should be able to see compiler errors, test failures, and other feedback and react to them.

Here I want a clean baseline first.

Quality isn't enough for a local model

Once the six quality tests are complete, there is another side to the benchmark.

The model has to run on my computer.

A model might produce excellent answers but take a minute to answer every prompt. Another might respond quickly but push macOS into heavy swap. A third might technically fit in memory while making IntelliJ and Chrome unpleasant to use.

So I added a second set of tests focused entirely on the experience.

Test 7: Measure a cold start

First, I make sure the model isn't already loaded:

ollama ps

If necessary:

ollama stop <model>

Then:

time ollama run <model> \
  "Write a Java method that returns the sum of two integers."

The useful measurement here is real: the elapsed time from starting the command until the response finishes.

This is the cold start because the model wasn't already loaded.

Test 8: Measure a warm response

Immediately run exactly the same command again:

time ollama run <model> \
  "Write a Java method that returns the sum of two integers."

Now the model should already be loaded.

For actual development, this measurement may be more important than the cold start. I don't normally load a model, ask one question, shut it down, and repeat.

A coding session looks more like:

Prompt → response
Prompt → response
Prompt → response

Warm performance is closer to that experience.

Test 9: Use a more realistic coding prompt

Adding two integers is useful for measuring startup overhead, but it doesn't represent much real work.

So I use a slightly more realistic prompt:

time ollama run <model> <<'EOF'
Create a Spring Boot service method that receives a customer ID,
retrieves the customer from a repository, and returns Optional<Customer>.
Use constructor injection and assume the repository has a findById method.
EOF

I'm not rescoring the quality here. The earlier tests already cover that.

I want to see how response time changes when the model needs to produce something closer to a normal development answer.

Test 10: Keep using the computer

This test doesn't need a script.

While the model is generating, I continue using the Mac.

I switch applications, type in another window, scroll Chrome, use IntelliJ, and keep the applications that would normally be running during development open.

Then I classify the experience:

Grade	Experience
A	Completely comfortable
B	Slight slowdown
C	Noticeable slowdown
D	Uncomfortable

A benchmark performed on an empty computer tells me how fast the model can run under ideal conditions.

I'm more interested in whether it works while the computer is doing its actual job.

Test 11: Repeat the same response

One measurement can also be misleading.

For a basic consistency check, I run the same prompt three times:

time ollama run <model> \
  "Explain the difference between an interface and an abstract class in Java."

Then I record:

Run 1:
Run 2:
Run 3:

I'm not trying to produce a statistically rigorous performance benchmark from three samples.

I just want to know whether the experience is reasonably consistent.

Something like:

5.1s
5.4s
5.0s

tells a different story from:

5.1s
14.8s
8.3s

The second result gives me something worth investigating before drawing conclusions from a single number.

Test 12: Watch memory, not only speed

My Mac has 16 GB of unified memory, shared by macOS, my development tools, and the model.

So while the model is generating, I also watch Activity Monitor.

I'm mainly interested in:

• Memory Used
• Memory Pressure
• Swap Used
• whether the system remains responsive

Swap means macOS is using storage to help compensate for memory pressure. Some swap is normal, but if running the model causes heavy swapping and makes everything else slower, that matters to the experiment.

I record the observations in plain language:

Memory pressure: Green
Swap: Minimal
System remained responsive: Yes

I don't need a laboratory full of monitoring tools to answer the practical question:

Can I comfortably run this model while I'm also developing software?

Keep the environment consistent

There is one final detail that can easily distort the comparison.

If I test one model with IntelliJ, Chrome, VS Code, and music running, then close everything before testing the next model, I've changed the environment.

My goal is to keep approximately the same normal workload running for every model.

I also don't optimize models individually during this benchmark.

No changing quantization between tests. No custom context tuning. No special Ollama configuration because one model happens to struggle.

The basic rule is:

Change the model, not the experiment.

There will always be some noise in a benchmark running on a normal computer. That's fine. I'm not publishing hardware laboratory results.

I'm trying to make the comparison repeatable enough that the differences mean something to me.

What I record

At the end, the journal for each model can remain surprisingly small:

Measurement	Result
Quality score
Cold start
Warm response
Realistic coding prompt
Repeated run 1
Repeated run 2
Repeated run 3
Mac responsiveness
Memory pressure
Swap usage
Overall perceived speed

Then I add the part that's harder to represent with numbers:

Would I comfortably use this model while developing?
Yes / Maybe / No

Would the response speed bother me?
Yes / No

Would I trust it to generate simple Java code?
Yes / No

Would I trust it to modify existing code without review?
Yes / No

Would I trust it to make architectural decisions?
Yes / No

These questions are subjective, but that doesn't make them useless.

I'm benchmarking a development tool I intend to use every day. Developer experience is part of the result.

The benchmark still doesn't tell me which model is the best agent

Even after all these tests, there is an important limitation.

The models are still working alone.

They haven't inspected a repository, selected relevant files, modified existing code, executed commands, run tests, observed failures, or tried to correct their own work.

So this benchmark can't answer:

Which model is the best coding agent?

It answers something narrower:

How capable is each model before I give it a project, tools, and an agent loop?

That's exactly the baseline I need.

Now I can run the same experiment across different models without changing the rules halfway through — and then compare what actually happens.