run · rs_1ux88er2dvh18f6n · graded 2026-09-17 19:41:10graded on sitepublished · anyone who can see the task can open this
Share

jev-preroute-30

claude-haiku-4-5 + claude-opus-5 + jev-1.13.0via jev-preroute-30on ⚡ Jev as a router: cheaper than always-Opus?

Grader output

free-form JSON · this task's own · run here
0.705grader.score141 of 200 cases passed
score
0.705
passed
true
total cases
200
pass threshold
0
n_scored=200n_correct=141by_category={"law":0.59,"engineering":0.82}n_escalated=43n_timed_out=0n_unanswered=0cost_usd_total=1.261latency_ms_p95=12658n_cost_metered=200n_cost_reported=200latency_ms_median=5724cost_unmetered_usd=0.005605cost_usd_per_correct=0.00898cases_accuracy_ci_low=0.655cases_escalation_rate=0.215n_escalation_reported=200cases_accuracy_ci_high=0.76n_first_answer_reported=0

The task's own judge and grader ran on this site over the answers the agent submitted. This task's reference answers are not published; the site holds them and never sends them to a solver. Which model answered, what it cost and who typed remain the client's word.

Run frame

declared by the client · not validated
cases
200
duration total · client wall clock
1438.26s
cost · client-declared
$1.261
engine
claude-haiku-4-5 + claude-opus-5 + jev-1.13.0
answers
via jev-preroute-30

run configuration

declared by the client · not validated
agent
jev-preroute-30
launcher
tp 0.0.15.dev29+gab6189d
executor
tp 0.0.15.dev29+gab6189d
framework
jev-preroute-30
model · declared
claude-haiku-4-5 (trap.yaml) · claude-opus-5 (trap.yaml) · jev-1.13.0 (trap.yaml)
solution
https://github.com/Ruqii/trapstreet-solutions@8d9f432e8d2f/model_routing/jev-preroute-30
task
https://github.com/trapstreet/trapstreet-tasks@7f36c68b831f/tasks/model_routing
trap
0.0.15.dev29+gab6189d

environment · self-reported, not validated

os
macOS 14.0
arch
arm64
kernel
Darwin 23.0.0
cpu
Apple M1 · 8 cores
memory
8.0 GB
python
3.13.7

skills & tools used

not reported

data coverage

identity
collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-17 19:41:08
model
collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-17 19:41:08
environment
collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-17 19:41:08
reproducibility
collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-17 19:41:08
skills
unsupported · tp runs a solver process; skills are a harness concept
tools
unsupported · tp does not observe the solver's tool calls
timing
collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-17 19:41:08
usage
collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-17 19:41:08 · $1.261 · reported
cases in this runcase_001: failcase_002: passcase_003: passcase_004: passcase_005: passcase_006: passcase_007: passcase_008: failcase_009: failcase_010: passcase_011: passcase_012: passcase_013: passcase_014: passcase_015: passcase_016: passcase_017: failcase_018: passcase_019: passcase_020: passcase_021: passcase_022: failcase_023: passcase_024: passcase_025: passcase_026: passcase_027: failcase_028: failcase_029: passcase_030: passcase_031: passcase_032: passcase_033: failcase_034: passcase_035: failcase_036: failcase_037: passcase_038: failcase_039: failcase_040: failcase_041: failcase_042: failcase_043: failcase_044: failcase_045: passcase_046: failcase_047: passcase_048: failcase_049: passcase_050: passcase_051: passcase_052: failcase_053: passcase_054: passcase_055: passcase_056: passcase_057: passcase_058: passcase_059: passcase_060: failcase_061: passcase_062: failcase_063: failcase_064: passcase_065: failcase_066: passcase_067: passcase_068: failcase_069: passcase_070: passcase_071: passcase_072: passcase_073: passcase_074: passcase_075: passcase_076: passcase_077: passcase_078: passcase_079: passcase_080: passcase_081: passcase_082: failcase_083: passcase_084: failcase_085: passcase_086: passcase_087: passcase_088: passcase_089: failcase_090: failcase_091: passcase_092: passcase_093: passcase_094: passcase_095: failcase_096: failcase_097: passcase_098: passcase_099: failcase_100: passcase_101: passcase_102: failcase_103: passcase_104: failcase_105: passcase_106: failcase_107: passcase_108: passcase_109: passcase_110: passcase_111: passcase_112: passcase_113: passcase_114: passcase_115: failcase_116: passcase_117: failcase_118: failcase_119: passcase_120: passcase_121: passcase_122: passcase_123: failcase_124: passcase_125: passcase_126: passcase_127: passcase_128: passcase_129: passcase_130: passcase_131: passcase_132: passcase_133: failcase_134: passcase_135: passcase_136: passcase_137: passcase_138: passcase_139: passcase_140: passcase_141: passcase_142: passcase_143: passcase_144: passcase_145: passcase_146: passcase_147: passcase_148: passcase_149: passcase_150: failcase_151: failcase_152: failcase_153: passcase_154: passcase_155: passcase_156: failcase_157: failcase_158: passcase_159: passcase_160: passcase_161: failcase_162: passcase_163: failcase_164: failcase_165: passcase_166: passcase_167: passcase_168: failcase_169: failcase_170: passcase_171: failcase_172: failcase_173: passcase_174: passcase_175: passcase_176: passcase_177: passcase_178: failcase_179: passcase_180: passcase_181: passcase_182: passcase_183: passcase_184: failcase_185: passcase_186: passcase_187: passcase_188: failcase_189: passcase_190: failcase_191: passcase_192: failcase_193: passcase_194: failcase_195: passcase_196: passcase_197: failcase_198: passcase_199: passcase_200: pass
per-case results · all 200, in case_id order
discuss this task →