Qwen vs Ministral vs GPT-OSS on an M2 Mac
I had the benchmark. Now I needed some models.
My goal wasn't to find the smartest local model available. With an M2 and 16 GB of unified memory, plenty of models aren't realistic options in the first place.
I wanted to find the best balance between:
- coding quality;
- reasoning;
- instruction following;
- response time;
- memory usage;
- normal development usability.
I selected four models that represented different points in this tradeoff:
| Model | Why I tested it |
|---|---|
| Qwen2.5-Coder 7B | Small coding-focused baseline |
| Qwen2.5-Coder 14B | Larger version of the same coding model |
| Ministral 3 8B | Small model designed with tool/agent use in mind |
| gpt-oss 20B | Larger model focused more heavily on reasoning |
I kept my normal workload running
I didn't close everything on the Mac before running the performance tests.
My normal workload remained active:
IntelliJ + running project
VS Code + running project
Chrome
Activity Monitor
Mail
Calendar
YouTube Music
Ollama
This isn't an ideal environment for producing clean hardware benchmark numbers.
It is, however, the environment where I actually want to use the model.
I'm not trying to turn my Mac into a dedicated inference server. I want the model running beside my development tools.
Other local-model experiments have reached a similar practical distinction: a model fitting in unified memory doesn't necessarily mean enough memory remains for a healthy workstation workload.
With the environment ready, I ran the same quality and performance tests from the previous post against all four models.
Basic coding: not much separation
The first test was intentionally simple:
Write a Java method that receives a List<Integer> and returns the first
duplicated value. Return Optional.empty() if there are no duplicates.
The solution should run in O(n) time.
All four models handled straightforward Java generation well.
| Model | Basic Java |
|---|---|
| Qwen 7B | 🟢 Very good |
| Qwen 14B | 🟢 Very good |
| Ministral 8B | 🟢 Very good |
| gpt-oss 20B | 🟢 Excellent |
This was my first useful result precisely because it wasn't very exciting.
If my benchmark only contained small coding exercises, I could probably conclude that all four models were great.
But gpt-oss also demonstrated why I needed to inspect answers rather than only the main algorithm. It voluntarily added support for null, then used Optional.of() in a path that could receive null.
The requested duplicate algorithm was correct. The extra behavior it invented wasn't.
Even the strongest model in the comparison could produce code that looked reasonable while hiding a subtle defect.
Code understanding exposed more differences
The second test gave the models an existing BigDecimal calculation and asked them to explain its behavior, assumptions, and edge cases.
All four understood the basic code.
The differences appeared in the details.
Some models identified reasonable concerns but also started inventing questionable edge cases that weren't supported by the Java types. Ministral, for example, discussed cases such as non-numeric values reaching a BigDecimal field and raised an integer-overflow concern that didn't match the calculation being performed.
This matters because more explanation isn't necessarily better reasoning.
| Model | Code understanding |
|---|---|
| Qwen 7B | 🟢 Good |
| Qwen 14B | 🟢 Good |
| Ministral 8B | 🟡 Good direction, some questionable reasoning |
| gpt-oss 20B | 🟢 Very good |
The responses were all useful, but I was starting to see where smaller technical inaccuracies could appear.
Debugging was easy for everyone
The debugging test contained a basic NullPointerException:
counts.put(word, counts.get(word) + 1);
Every model correctly understood that Map.get() returns null for a key that hasn't been inserted yet and that Java then attempts to unbox the value during + 1.
| Model | Debugging |
|---|---|
| Qwen 7B | 🟢 |
| Qwen 14B | 🟢 |
| Ministral 8B | 🟢 |
| gpt-oss 20B | 🟢 |
Again, not much separation.
For small, isolated Java problems, even the smaller models were already quite capable.
The comparison became more useful when I moved away from syntax and debugging.
Reasoning finally separated the models
The reasoning prompt included an important constraint:
You need to process 10 million transaction IDs and determine which IDs
appear more than once.
Memory is limited, and the transaction IDs are integers.
Describe two possible approaches, including their time and space complexity,
and explain when you would choose each approach.
Finding duplicates isn't difficult.
Reasoning about 10 million values under a memory constraint is the actual problem.
The Qwen models proposed reasonable approaches but weren't as strong at connecting the solutions back to the memory constraint.
Ministral explored sorting and external sorting, which moved in a useful direction, but some of its complexity analysis and implementation details were inconsistent.
gpt-oss gave the strongest response here.
| Model | Algorithm / constraint reasoning |
|---|---|
| Qwen 7B | 🟡 |
| Qwen 14B | 🟡 |
| Ministral 8B | 🟡 Broader exploration, but technical mistakes |
| gpt-oss 20B | 🟢 Strongest |
This was the first test where the larger reasoning model showed a meaningful advantage.
Architecture produced a similar pattern
The Spring Boot architecture test asked where Redis cache orchestration should live in:
Controller → Service → Repository → PostgreSQL
↕
Redis
The models needed to discuss the responsibilities of the Controller, Service, Repository, and cache layer.
gpt-oss produced the strongest architectural reasoning.
Qwen 14B also performed reasonably well. Qwen 7B showed more architectural weakness, while Ministral gave a generally sensible answer but omitted the Repository responsibility even though the prompt explicitly requested it.
| Model | Architecture reasoning |
|---|---|
| Qwen 7B | 🟡 |
| Qwen 14B | 🟢 Good |
| Ministral 8B | 🟡 Incomplete |
| gpt-oss 20B | 🟢 Strongest |
At this point, gpt-oss was clearly demonstrating why I included a larger model.
But I hadn't looked at performance yet.
Instruction following produced an unexpected miss
For the final quality test, the models had to solve balanced parentheses while respecting several constraints.
One requirement was deliberately boring:
Provide the solution first.
Then provide exactly three short bullet points explaining it.
Qwen 7B followed it.
Ministral followed it.
gpt-oss followed it.
Qwen 14B returned five bullets.
| Model | Instruction following |
|---|---|
| Qwen 7B | 🟢 |
| Qwen 14B | 🟡 Missed explicit constraint |
| Ministral 8B | 🟢 |
| gpt-oss 20B | 🟢 |
The code itself was correct.
For normal chat, I wouldn't care much about two additional bullets.
For an agent, I care considerably more.
If the instruction is eventually:
Do not modify the Repository.
I need the model to treat it as a requirement rather than a suggestion.
The quality scores were surprisingly close
After scoring correctness, reasoning, and instruction following across the six tests, I had:
| Model | Score | Percentage |
|---|---|---|
| gpt-oss 20B | 35/36 | 97.2% |
| Qwen2.5-Coder 7B | 32/36 | 88.9% |
| Qwen2.5-Coder 14B | 32/36 | 88.9% |
| Ministral 3 8B | 30/36 | 83.3% |
The result I found most interesting wasn't gpt-oss finishing with the highest score.
It was this:
Qwen 7B → 32/36
Qwen 14B → 32/36
Doubling the parameter count didn't improve the final score in my tests.
This is a small custom benchmark, so I wouldn't use it to claim the 7B model is generally as capable as the 14B model. It tells me something narrower and more useful: for the tasks I selected, on this experiment, the larger model didn't provide a measurable quality improvement.
Now the additional hardware cost had to justify itself elsewhere.
Speed changed the comparison
The performance tests made the tradeoff much clearer.
| Model | Cold | Warm | Spring Boot prompt | Longer response* |
|---|---|---|---|---|
| Qwen 7B | ~8 s | ~8–10 s | — | — |
| Qwen 14B | 31.0 s | 19.9 s | 58.0 s | ~58 s avg |
| Ministral 8B | 19.9 s | 12.0 s | 21.3 s | ~63 s avg |
| gpt-oss 20B | 72.1 s | 52.0 s | 80.2 s | ~219 s avg |
*Three repeated runs explaining the difference between a Java interface and an abstract class.
gpt-oss had produced the best quality score.
But a warm response to a trivial prompt took around 52 seconds.
Its longer explanation averaged around 219 seconds — more than three and a half minutes.
For a difficult question I ask occasionally, I can wait.
For an interactive coding agent, this becomes much harder to accept.
One development task might require several model calls.
A 50-second response isn't necessarily a 50-second task anymore.
Verbosity also costs time
gpt-oss had another characteristic that became more noticeable during the performance tests: it liked to explain things.
A lot.
Sometimes I asked for a small method and received the method, class structure, examples, explanation, alternatives, and additional context.
The information wasn't necessarily bad. But locally, every additional generated token takes time.
This means practical latency depends on more than model loading and raw inference speed:
Response time
=
Model performance
+
Amount of reasoning
+
Amount of generated output
A more capable model can still feel slower partly because it produces considerably more than I need.
For agentic coding, concise and correct may be more useful than comprehensive and slow.
Memory completed the picture
The Mac has 16 GB of unified memory.
With my normal development workload active, the difference between the models became visible in Activity Monitor:
| Model | Memory behavior | Mac experience |
|---|---|---|
| Qwen 7B | Comfortable | 🟢 Best responsiveness |
| Ministral 8B | ~13 GB, green pressure, minimal swap | 🟢 Comfortable |
| Qwen 14B | ~15 GB, yellow pressure, ~4 GB swap | 🟡 Noticeably heavier |
| gpt-oss 20B | ~15.5 GB, yellow pressure, ~6 GB swap | 🟠Heavy |
All models returned to roughly 10 GB and green memory pressure when idle.
The interesting part was what happened while generating.
Ministral stayed around 13 GB with green memory pressure and minimal swap. I could continue using the computer normally.
Qwen 14B pushed the machine toward yellow memory pressure and around 4 GB of swap.
gpt-oss reached approximately 15.5 GB of memory usage and around 6 GB of swap. The Mac remained mostly usable, but YouTube Music couldn't play normally while the model was responding.
That's a very practical benchmark result.
I don't particularly care whether a model technically fits into 16 GB.
I care whether I can keep coding while it runs.
Bigger wasn't automatically better
The four models ended up occupying different positions:
Qwen 7B
↓
Good quality
Fast
Comfortable
Ministral 8B
↓
Slightly weaker standalone score
Good performance
Comfortable
Qwen 14B
↓
Good quality
Slower
More memory pressure
gpt-oss 20B
↓
Best reasoning
Very slow
Heavy memory pressure
This is why a single quality score wouldn't have been enough.
gpt-oss clearly produced the strongest standalone reasoning in my tests. If I were choosing a model only for difficult questions where I don't care about waiting, it would be very interesting.
But I'm building an interactive coding environment.
Latency gets multiplied by the agent loop. Memory pressure competes directly with IntelliJ, VS Code, browsers, databases, and everything else I'm already running.
The smallest model suddenly becomes much more competitive.
My decision before agent testing
Based only on these standalone tests, I wouldn't choose one model for every possible task.
I would give them different roles.
Qwen2.5-Coder 7B became my default local candidate. It gave me the best overall combination of quality, speed, and resource usage.
Ministral 3 8B became the main challenger. Its standalone score was slightly lower, but it remained fast and comfortable on the Mac, and its tool/agent capabilities made it particularly interesting for the next stage.
gpt-oss 20B became the reasoning fallback. It produced the best quality, but the latency and memory cost were too high for me to treat it as the default interactive model.
Qwen2.5-Coder 14B ended up in an awkward middle position. It matched the 7B model's 32/36 score while consuming more memory and taking considerably longer to respond.
The result wasn't what I expected when I started looking at parameter counts.
Final comparison
Here is the complete picture before moving into the actual coding-agent experiments:
| Result | Qwen2.5-Coder 7B | Ministral 3 8B | Qwen2.5-Coder 14B | gpt-oss 20B |
|---|---|---|---|---|
| Quality | 32/36 (88.9%) | 30/36 (83.3%) | 32/36 (88.9%) | 35/36 (97.2%) |
| Reasoning | 🟡 Good | 🟡 Good, some technical mistakes | 🟢 Good | 🟢 Best |
| Speed | 🟢 Fastest | 🟢 Fast | 🟡 Slow | 🔴 Very slow |
| Memory | 🟢 Comfortable | 🟢 ~13 GB / green | 🟡 ~15 GB / ~4 GB swap | 🟠~15.5 GB / ~6 GB swap |
| Normal Mac use | 🟢 Comfortable | 🟢 Comfortable | 🟡 Noticeable pressure | 🟠Interfered with normal workload |
| Decision | Default candidate | Main challenger | Secondary candidate | Reasoning fallback |
For my M2 with 16 GB of unified memory, the 7–8B range looked like the practical sweet spot.
The larger model demonstrated something important: there was more reasoning capability available if I was willing to pay for it in time and memory.
But local coding isn't only about getting the best answer eventually.
I need the model to work with me, while the rest of my development environment is running.
So I started the next phase with Qwen 7B and Ministral 8B as the most interesting candidates.
And there was still one major assumption left to test:
A model that performs well alone isn't necessarily the model that performs best when an agent starts giving it tools.