all tasks / Jev-class: name the weakness class / jev-1.13.0 / rs_aobcib9q29y6qpwz

run · rs_aobcib9q29y6qpwz · graded 2026-09-20 17:56:34graded on sitepublished · anyone who can see the task can open this

jev-1.13.0

jev-1.13.0via tp 0.0.15.dev29+gab6189don Jev-class: name the weakness class

Grader output

free-form JSON · this task's own · run here

0.745grader.score1118 of 1500 cases passed

score: 0.7453

n_cases=1500n_errors=382n_scored=1500n_correct=1118by_category={"CWE-20":0.76,"CWE-22":0.92,"CWE-78":0.94,"CWE-79":0.98,"CWE-89":0.98,"CWE-94"…n_timed_out=0n_unanswered=0n_worst_band=854cases_accuracy=0.7453latency_ms_p95=1697n_cost_metered=0latency_ms_median=775n_with_confidence=1500cost_unmetered_usd=0.071266cases_worst_band_lo=0.9cases_overconfidence=0.0618cases_worst_band_gap=0.051cost_usd_per_correct=0.000064cases_mean_confidence=0.8072cases_unrelated_share=0.6597n_parent_child_errors=130cases_calibration_error=0.0618cases_parent_child_share=0.3403cost_usd_per_thousand_correct=0.0637

The task's own judge and grader ran on this site over the answers the agent submitted. This task's reference answers are not published; the site holds them and never sends them to a solver. Which model answered, what it cost and who typed remain the client's word.

Run frame

declared by the client · not validated

cases: 1500
duration total · client wall clock: 2936.41s
cost: unknown · not reported
engine: jev-1.13.0
answers: via tp 0.0.15.dev29+gab6189d

run configuration

declared by the client · not validated

launcher: tp 0.0.15.dev29+gab6189d
executor: tp 0.0.15.dev29+gab6189d
framework: jev
model · declared: jev-1.13.0 (trap.yaml)
solution: https://github.com/Ruqii/trapstreet-solutions@ba65a005ca2c/cve_weakness_class/jev
task: https://github.com/trapstreet/decision-layer-bench@0454940c1a0e/tasks/cve_weakness_class
trap: 0.0.15.dev29+gab6189d

environment · self-reported, not validated

os: macOS 14.0
arch: arm64
kernel: Darwin 23.0.0
cpu: Apple M1 · 8 cores
memory: 8.0 GB
python: 3.13.7

skills & tools used

not reported

data coverage

identity: collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-20 17:43:31
model: collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-20 17:43:31
environment: collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-20 17:43:31
reproducibility: collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-20 17:43:31
skills: unsupported · tp runs a solver process; skills are a harness concept
tools: unsupported · tp does not observe the solver's tool calls
timing: collected · tp · tp/0.0.15.dev29+gab6189d · 2026-09-20 17:43:31
usage: not yet reported · cost unknown · not recorded

cases in this run

per-case results · all 1500, in case_id order

casesolutiondurationcostjudge

discuss this task →