BENCH: The Smaller Model Won

I stopped judging an AI skill by vibes and put it on the bench instead.

I had a skill that looked useful.

When I ran it manually, the answers felt more useful. The agent asked better questions, challenged weak plans, and identified documentation gaps. It behaved more like a careful reviewer than an overconfident code generator.

That felt promising. But “felt promising” is a dangerous place to stop, because manual AI testing is generous.

You remember the best answers and overlook the poor ones. You slightly adjust the prompt and give the model another chance. You compare today’s answer with what you remember from yesterday. Before long, you stop focusing on the skill.

You are evaluating your patience.

So I wanted to answer a better question:

Can I evaluate an AI skill the same way I would evaluate code?

Not vibes. Not “it looked good in the chat.” Not one-off manual testing.

Something repeatable.

I tested a skill called “Roast the Docs.” The skill helps agents evaluate implementation plans against project documentation. Before an agent writes code, it should check the glossary, context maps, existing Architectural Decision Records (ADRs), and current plans. The point is to help the agent ask clearer questions, use project language consistently, spot inconsistencies, and suggest documentation updates when important decisions appear.

That sounds helpful, but helpful is not the same as reliable. So I built a small evaluation harness, and the result surprised me.

The harness changed the question from:

Which model feels smarter?

To:

Which model performs better for this skill?

That became the shape of the work: BENCH.

• B - Baseline the task
• E - Evaluate real outputs
• N - Notice failure modes
• C - Compare models honestly
• H - Hold the lesson

B - Baseline the task

The first thing I needed was a baseline.

Before this, testing a skill meant running a prompt and deciding whether the answer looked good. That is fine at first, but it is weak once the skill matters.

The problem is that too many things can change at once: the workspace, prompt, model, expectations, and even your mood.

When the output looks better, it can be difficult to determine what actually improved. Was it the skill, the model, the prompt, the example, or just luck?

So I defined what “good” meant before the model touched the task.

The baseline was simple:

1. Run the agent without the skill.
2. Run the agent with the skill.
3. Give both runs the same workspace.
4. Grade both runs against the same expected behaviours.

Good was not “the answer sounds polished.”

Good was:

• Did it find the existing ADR instead of creating a duplicate?
• Did it ask a clarifying question when the terminology was unresolved?
• Did it preserve the existing domain language?
• Did it avoid overengineering a reversible change?
• Did it notice when code and documentation disagreed?

That gave the harness something concrete to score.

E - Evaluate real outputs

The harness runs the same set of test cases in two modes:

1. Agent with the skill enabled
2. Agent without the skill enabled

Each test case gives the agent a small workspace. This can include a specific scenario, an ADR, a glossary entry, a plan, or a minor conflict between code and documentation.

Some cases are intentionally awkward. A plan may look architectural but be easy to reverse. A small wording choice may create a long-lasting boundary. Existing documentation may already answer the question, but only if the agent bothers to read it.

The goal was not to reward a fluent answer. The goal was to see whether the skill changed behaviour in situations where the wrong move would matter.

A test case looked roughly like this:

Test case
  |-- Workspace files
  |-- Skill on/off
  |-- Model configuration
  |-- Expected behaviours
  `-- Grader checks

One test case involved an existing ADR for an API boundary. The implementation plan suggested creating a new ADR for the same decision.

Without the right behaviour, the agent treated it as a fresh decision.

With the skill, the better answer was to recognise the existing ADR and either reference it or ask whether the plan intended to change that decision.

That is the difference between judging a demo and evaluating a workflow.

N - Notice failure modes

The harness was useful because it made failures specific.

Spark did better on cases like:

• detecting ADR-worthy tradeoffs
• avoiding duplicate ADRs
• resisting overengineering pressure
• skipping ADRs for obvious or reversible changes
• preserving existing terminology

It was not perfect. GPT-5.5 did better in a couple of unclear situations where multiple contexts could be interpreted in different ways. That mattered too.

A benchmark should not only show that your idea works. It should show where your confidence runs out.

That is the useful part of failure.

The failure modes told me what to add next:

• messy plans
• stale documentation
• conflicting terminology
• existing ADRs that should not be duplicated
• decisions that look architectural but are actually reversible
• documentation that disagrees with code
• project language that should be preserved, not renamed

The benchmark gets more useful as it gets more annoying.

Easy cases make demos look good.

Annoying cases make tools better.

C - Compare models honestly

Once the harness existed, I could run the same tests against different models.

That shifted the question again:

Which model passes more checks, uses fewer tokens, and finishes faster on the same test suite?

The two runs I compared were:

• GPT-5.5 medium
• GPT-5.3 Codex Spark xhigh

I expected the bigger model to win.

It did not.

The smaller model was not just cheaper or faster. It produced the better result for this skill.

Using the skill:

Spark was 7.81 percentage points more accurate, used around 19,409 fewer tokens per evaluation, and finished about 14 seconds faster on average.

That was the surprise.

The larger model often sounded polished. The smaller model stayed closer to the instructions. In this harness, that mattered more than raw capability.

Here is the comparison that changed my mind:

This is not a universal benchmark. It is one skill, one dataset, and one set of behaviours I cared about.

But that is exactly why it was useful.

I did not need a model leaderboard. I needed to know which model performed better for this workflow.

H - Hold the lesson

The real win was not that Spark performed well.

The real win was that I did not have to guess.

Without the harness, I probably would have chosen the larger model. That feels safer. Bigger usually suggests more reasoning, more capability, and fewer surprises.

The harness showed a better answer: for this skill, Spark was clearer, faster, cheaper to run, and more accurate against the behaviours I cared about.

That matters because AI cost is not just the price per request.

It is the price per useful result.

A cheaper model is not better if it fails more often, needs extra review, or misses the behaviour the workflow requires. But if a smaller model passes more checks and uses fewer tokens, it can quickly become the better engineering choice.

Once the harness existed, I could:

• Add tricky edge cases
• Run the same tests against multiple models
• Compare skill-enabled and skill-disabled runs
• Measure accuracy, token usage, and latency
• Keep timestamped results for future comparison
• Make model selection a cost and quality decision instead of a guess

TL;DR version

Final takeaway

If you remember one thing, make it this:

The best model is the one that performs best for your workflow, under the constraints you can measure.

The smaller model did not win because it was smarter.

It won because the task was clear, the scoring was consistent, and the harness rewarded the behaviour I actually needed.

That was the real lesson.

In AI work, benchmarking is not a side quest. It is how you stop guessing.