Concepts

The mental model behind the leaderboard — tasks, solutions, runs, and how scoring, ranking, and trust actually work.

The mental model behind the leaderboard. Read this once and the rest of the docs (and the UI) stop being surprising.

The five nouns

Task — a benchmark: inputs, expected answers, and a judge. Has a stable slug (/tasks/<id>); the scored cohort is one version (a pinned task-repo commit).
Solution — a program under test, identified by its content (repo + commit, or a content hash when local). Shared and unowned: two people who submit the same commit land on the same solution.
Run — one execution of a solution against a task version. Carries a score, cost, latency, and per-case results. A run row exists only because it was scored and uploaded — there is no "failed run" state.
Case — one input/expected pair inside a task. The judge scores each case; cases.metrics holds whatever the judge emitted (answer, expected, reason…).
Version — tasks are versioned by commit. Runs attach to a version, so a board always compares solutions that ran the same task code.

trap treats your solution as a black box and hands each side directory paths in a JSON manifest — it never inlines file contents:

Who	Env var	Gets
solution	`TRAP_MANIFEST`	`{inputs_dir, outputs_dir}` — read inputs, write your answer
judge	`TRAPTASK_MANIFEST`	`{inputs_dir, expected_dir, outputs_dir, run}` — scores one case
grader	`TRAPTASK_MANIFEST`	the list of case results — writes the run summary

Solution repo and task repo stay fully decoupled; they share only this contract. → build a solution · build a task.

A run's score is the grader's score if it emitted one, otherwise the mean of case scores. A case counts as passed when its score is 1.0 — that's a display convention for binary tasks, not a run-level verdict.
The board is not a single number. Each eligible user contributes their median score across their own runs of a solution; the solution ranks by the median of those medians, tie-broken by reproduction count (how many distinct users ran it), then cost, then latency.
A solution with fewer than 3 reproducers is provisional — shown, flagged, not yet trusted.

Every submit carries provenance: the solution's repo + commit and the task's repo + commit — the minimal seed to reproduce the run. The server locates the task by content, never by a typed-in name.
Local solutions (no public repo) are welcome as a try-out but are constructively unranked — they never enter any median. Push to a public repo and resubmit to rank.
Today's tier is self-reported: you run it, we record what you upload (cost and latency included). A verified tier — we re-run in a sandbox with held-out inputs — is reserved for later.