Build a task

End-to-end: design a task, write the solution, run it locally, register it on trapstreet, submit.

We're going to build a task called sum-two-numbers. Solutions get two ints in a JSON file. They write a program that adds them and writes the sum to another JSON file. We score whether their answer matches ours. Whole thing takes about 15 minutes.

What you're making

A task is a folder. Four things live in it:

• inputs/<case>/ — what we hand the solution
• expected/<case>/ — what we expected back
• judge.py — scores one case
• grader.py — aggregates case scores into a run-level summary

Plus a traptask.yaml that wires them together. The solution and your task only ever talk through directories and environment variables — trap hands each side absolute directory paths in a JSON manifest and never inlines file contents.

Step 1 — make the case files

mkdir -p sum-task/inputs/basic     sum-task/expected/basic
mkdir -p sum-task/inputs/negatives sum-task/expected/negatives
mkdir -p sum-task/inputs/zero      sum-task/expected/zero

Inputs:

echo '{"a":  3, "b":  5}' > sum-task/inputs/basic/nums.json
echo '{"a": -1, "b": -2}' > sum-task/inputs/negatives/nums.json
echo '{"a":  0, "b":  0}' > sum-task/inputs/zero/nums.json

Expected outputs:

echo '{"sum":  8}' > sum-task/expected/basic/sum.json
echo '{"sum": -3}' > sum-task/expected/negatives/sum.json
echo '{"sum":  0}' > sum-task/expected/zero/sum.json

The folder names under inputs/ and expected/ are the case ids.

Step 2 — write the judge

judge.py runs once per case. trap injects TRAPTASK_MANIFEST — a JSON string of absolute directory paths. You read the files you need and print a JSON object with at least a numeric score.

# sum-task/judge.py
import json, os
from pathlib import Path

m = json.loads(os.environ["TRAPTASK_MANIFEST"])
# m = {
#   "inputs_dir":   ".../inputs/<case>",      # this case's inputs
#   "expected_dir": ".../expected/<case>",    # null if the case has no expected/
#   "outputs_dir":  ".../solution/outputs",   # ONLY the solution's written files
#   "run": {"stdout": <path>, "stderr": <path>, "meta": <path>},  # captured streams
# }

actual   = json.loads((Path(m["outputs_dir"])  / "sum.json").read_text())
expected = json.loads((Path(m["expected_dir"]) / "sum.json").read_text())

correct = actual.get("sum") == expected["sum"]
print(json.dumps({
    "score": 1.0 if correct else 0.0,
    "correct": correct,
}))

You join the directory paths with the filenames you authored. run.meta points at a JSON file {exit_code, duration} if you want the solution's exit status; run.stdout / run.stderr are its captured streams (many LLM solvers answer on stdout — read run.stdout instead of a file).

Step 3 — write the grader (or skip it)

grader.py runs once at the end. trap injects TRAPTASK_MANIFEST — here it's the JSON list of per-case results — and you print one run-level summary:

# sum-task/grader.py
import json, os

cases = json.loads(os.environ["TRAPTASK_MANIFEST"])   # [{case_id, metrics, ...}, ...]
scores = [c["metrics"]["score"] for c in cases if c.get("metrics")]
avg = sum(scores) / len(scores) if scores else 0
print(json.dumps({
    "score": round(avg, 3),
    "all_correct": all(s == 1.0 for s in scores),
}))

You can skip writing grader.py entirely. If it's missing, the server averages the case scores for you. Write your own to emit a different run-level score (say, all-or-nothing) or extra metrics — everything it prints is stored and shown on the run page, but only score affects ranking. There is no run-level pass/fail: a run row exists because it was scored, full stop.

Step 4 — wire it up with traptask.yaml

# sum-task/traptask.yaml
name: Sum Two Numbers          # optional: human-readable task title

dirs:                          # optional; these are the defaults
  inputs: inputs/
  expected: expected/

cases:
  - id: basic
  - id: negatives
  - id: zero

judge:
  cmd: uv run python judge.py

grader:
  cmd: uv run python grader.py

You also need a pyproject.toml next to traptask.yaml so uv run can build a venv for judge/grader:

[project]
name = "sum-task"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = []

That's the task. Push the sum-task/ folder up to a GitHub repo.

Step 5 — publish on trapstreet

Go to /tasks/new, paste your task's GitHub URL, review the prefilled values, hit Create. The CLI locates your task by its repo + commit (content-addressed), so any solver that points source: at this repo can run against it.

When you create the task as public, trapstreet requires every submitted solution to carry a publicly reachable git repo — see build a solution for how that flows from the solver side.

What you didn't have to think about

• Case orchestration — tp runs each case in its own subprocess, captures stdout/stderr/exit_code, handles timeouts.
• Result storage — .trap/<task>/<ts>/report.json is produced automatically, ready to upload.
• Leaderboard columns, ranking, dedup — the server reads score from your grader (or averages the case scores) and derives cost + latency from the report itself. Zero config; see the reference for the full metrics story.

Gotchas worth remembering

• Case ids are folder names. inputs/basic/ and expected/basic/ must match exactly.
• Manifest values are directories, not files. Join m["outputs_dir"] with the filename you expect — trap hands you locations, never inlined contents.
• judge.py output schema is yours. Whatever JSON keys you print flow through to cases.metrics; pick names you'll want in the per-case view. Include agent_answer / expected / reason and they render in the per-case failure view.
• grader.py is optional. Skip it unless you want a run-level score rule other than "average of case scores".