LLM evals
Can Jev Route Between Cheap and Expensive LLMs?
2026-09-18 · by Ruqi · 4 min read
Can a tiny decision model tell you when a cheap LLM is good enough — and when you need to pay for a stronger one?
I tested this with TypeSafe's new Jev model as a router between Haiku and Opus.
The result was surprisingly asymmetric: give Jev a Haiku answer and ask "is this wrong?" and you get 94% of Opus's accuracy for 54% of the cost. Ask it to predict "will Haiku get this wrong?" before Haiku answers, and it performs close to random.
How each setup works
I tested two ways of using Jev as a router, plus a random-escalation control and the two obvious baselines.
Cascade — answer first, check after.
jev-cascade-30(~30% of questions escalated)jev-cascade-60(~60% of questions escalated)
Preroute — guess the difficulty first.
jev-preroute-30(~21% of questions escalated)
Random-30 — the control. Instead of Jev deciding what to escalate, ~30% of questions are picked at random and re-answered by Opus.
Baselines: opus-only and haiku-only. no router, just one model answering everything.
All 6 ran on the same 200 MMLU-Pro questions (100 law, 100 engineering) — chosen because cheap and frontier models are unusually far apart on them.
All six runs, ranked by score — from the model-routing leaderboard, where every row links to its full per-case results.
The headline result
Cascade-30 gets 94% of Opus's accuracy at 54% of the cost. Preroute costs about the same, but its accuracy drops by 0.145.
Random escalation barely helps at all: 0.675 accuracy versus Haiku's 0.670, while costing 2.7x more.
Why preroute fails
This is the part worth actually reading.
Evidence 1 — Jev's score doesn't separate the two groups. I scored every question with Jev's confidence and checked how well that score separates "Haiku got this wrong" from "Haiku got this right" (AUC: 0.5 is random, 1.0 is perfect separation.)
| Question Jev is asked | Misses | Hits | AUC |
|---|---|---|---|
| "Is this answer wrong?" (cascade) | 0.58 | 0.34 | 0.79 |
| "Will the cheap model get this wrong?" (preroute) | 0.85 | 0.84 | 0.60 |
With cascade, Jev assigns noticeably higher scores to Haiku's misses than its hits. With preroute, the scores are almost identical: 0.85 vs. 0.84. Before seeing Haiku's answer, Jev has very little signal about which questions Haiku will actually miss.
Evidence 2 — hit rate. Haiku got 66 of the 200 questions wrong.
- Cascade escalated 58 questions. 42 of them were genuine Haiku mistakes — a 72% hit rate.
- Preroute escalated 43 questions. 17 of them were genuine mistakes — a 40% hit rate.
- Random-30 escalated 63 questions. 16 of them were genuine mistakes — a 26% hit rate.
(For reference, blindly picking questions would have an expected hit rate of 33%).
Cascade also did less damage when it escalated: only 2 correct Haiku answers became wrong after escalation to Opus, compared with 5 under random escalation.
Evidence 3 — preroute escalated the wrong subject. Haiku is weakest on law (0.62, versus 0.89 for Opus), so that's where escalation should help most. But preroute escalated only 8 law questions, while most of its escalation budget went to engineering.
The pattern across all three tests is pretty clear: Jev is much better at judging an answer it can see than predicting whether another model will get a question wrong before it answers.
Diminishing returns on escalation rate
More escalation doesn't necessarily mean a better trade-off.
Doubling the number of escalations recovers just 2 additional answers, while increasing cost by 79%. By cascade-60, the total cost is close to simply running Opus on everything, but accuracy is lower.
In eval, what really matters isn't whether it gets everything right. It's how much each additional point of accuracy costs.
So, it's important to set a clear acceptance threshold first, then decide whether those extra few points are worth paying twice as much for. Otherwise, "cheap" and "accurate" just stay as competing claims with no concrete way to choose between them.
What this doesn't show, on purpose:
- Every number here is a single run per setup — no repeats, no variance bars.
- MMLU-Pro is public; these models may have seen some of it during training.
- Opus costs about 5x Haiku here — a cheaper first-pass model would open more room to save.