Harness
Bigger Model or Better Harness?
2026-09-16 · by Ruqi · 2 min read
Here's the question I wanted to test: if you could only change one thing to make your agent perform better, would you use a better model, or a better harness?
I used a simplified version of DABStep: a year of credit card transactions, a fee manual, and 25 questions. 20 are hard: the agent has to understand and apply the rules, not just look up a number.
I tested three harnesses: claude-code, dsh, and mini-loop. mini-loop is a basic ReAct loop I wrote as the baseline.
Cheap models swing more. Expensive ones don't.
The difference is pretty clear. Deepseek-flash drops from 0.840 with a full harness to 0.480 with mini-loop — a 36-point drop. Claude-opus-5 only drops from 0.760 to 0.680.
The cheaper model seems much more dependent on the harness.
Score by setup, deepseek-flash vs. claude-opus-5. mini-loop is a bare custom loop — no real harness — run as the control.
With the right harness, DeepSeek gets the highest score
Deepseek-flash on claude-code harness scored 0.840 for $0.615. Opus on the same harness scored 0.760 for $8.126 — lower score, 13x the cost.
The trade-off is speed. DeepSeek took 5,258 seconds (88 mins), versus 1,520 (25 mins) for Opus.
And claude-code isn't actually the best harness for Opus. On dsh, it gets the same score at about the same speed, for 28% less cost. There's no single winner, it depends on whether you care most about score, cost, or speed.
Score, cost, and latency — deepseek-flash · claude-code vs. claude-opus-5 on claude-code and dsh.
Max effort didn't help Kimi here
Kimi-k3 on claude-code ran with reasoning effort set to max. It was the most expensive run ($14.436), the slowest (3.4 hours), and the only one that left five questions unanswered.
It still scored just 0.640. In this run, more compute didn't translate into better results.
Cost per run, all 8 graded runs — dabstep leaderboard.
The biggest takeaway is how much the harness can matter. DeepSeek went from the worst score in the benchmark to the best depending on what it ran with.
Opus was much more stable, but the harness still mattered: dsh matched its claude-code score at about the same speed, for 28% less cost.
More compute didn't guarantee a better result either. Kimi was the slowest and most expensive run, but still scored near the bottom.
One caveat: every run is reproducible. If you're curious, run the same setup yourself, or add a new model + harness combination to the leaderboard.
You can see all the runs on the DABStep leaderboard, or try your own combinations there.