← All tasks

Karpathy's jagged questions — the 50 m car wash

ranked by score ↓
Source

Paste as source: in your trap.yaml

git+https://github.com/xiaotianhan91/karpathys-jagged-questions@e13950d4882019936d16c6cf7b756b4d4fc68274
Share

karpathys-jagged-questions

A trap task built around an anecdote Andrej Karpathy told at the Sequoia AI Ascent fireside chat (April 2026): a state-of-the-art model can refactor a 100k-line codebase, yet advises you to walk to the car wash 50 meters away

1 case

Each case feeds files from inputs/<id>/ to the solution, expects files in expected/<id>/, and is scored by judge.py then aggregated by grader.py.

traptask.yaml · source on GitHub

cases (1)

▸car_wash_50m>-

input

question.txt

My car is dirty and I want to get it washed. There is a car wash just 50 meters from my
home. Should I walk there or drive? Start your answer with exactly one word — WALK or
DRIVE — then briefly explain why.

expected output

expected.json

{
  "verdict": "drive",
  "rationale": "The car itself must reach the car wash, so the only sane opening word is DRIVE. Answering WALK is the jagged-intelligence failure; any other opening word means the format instruction was ignored, which also fails."
}

Scored by judge.py — see Scoring logic below for the full rule.

scoring logic

judge.py runs once per case and prints a score per case. grader.py runs once at the end and folds case scores into a run-level summary. Without grader.py, the run's score is simply the average of case scores.

▸judge.py23 lines · view on GitHub
"""Per-case judge for the jagged-intelligence car-wash question.

The question instructs the solution to open its answer with exactly one word
— WALK or DRIVE — so the judge simply compares the first alphabetic word of
stdout (case-insensitive, skipping punctuation and markdown) with the
verdict in the case's ``expected.json``. Opening with anything else fails
too: ignoring the format instruction is its own failure.
"""

import json
import os
import re
from pathlib import Path

if __name__ == "__main__":
    manifest = json.loads(os.environ["TRAPTASK_MANIFEST"])
    answer = Path(manifest["run"]["stdout"]).read_text().lower()
    expected = json.loads((Path(manifest["expected_dir"]) / "expected.json").read_text())

    first_word = re.search(r"[a-z]+", answer)
    passed = first_word is not None and first_word.group() == expected["verdict"].lower()
    print(json.dumps({"passed": passed, "score": 1.0 if passed else 0.0}))
▸grader.py14 lines · view on GitHub
"""Overall grader: aggregates per-case judge scores into a single verdict."""

import json
import os

if __name__ == "__main__":
    results = json.loads(os.environ["TRAPTASK_MANIFEST"])
    scores = [r["metrics"]["score"] for r in results if r["metrics"]]

    if scores:
        print(json.dumps({"passed": all(s == 1.0 for s in scores), "score": sum(scores) / len(scores)}))
    else:
        print(json.dumps({"passed": False, "score": 0.0, "error": "no scored cases"}))