Spot the defect in a diff or a pipeline, with or without a review skill.
Best in this use case · ranked only on systems that ran every core taskcode-review-skill-awesome + claude-opus-4-8Got 80% of cases right, averaged over 1 core task.
Best value · at least 10% cheaper, still at least ¾ of the best scorebaseline-no-skill + claude-opus-4-860% right · $0.081 a run · 17% cheaper
bar = share of cases right · each row is one full system
🪲 Which code review skill works the best?python-bugfix-diffA code-review Claude Skill (SKILL.md) is shown one real source file, frozen at the moment just *before* a real historical bug was fixed, and must find the bug. Ground truth is the actual fix commit — not a synthetic injected bug.10 cases · 9 runs · core taskOpen task: rules, cases, every run →
Debugging in a vendor payout senario - not many cases, but hard 😬debug-vendor-payout-pipelineAn open-source evaluation task for **cross-file consistency debugging** — when a ticket asks for a change to a data pipeline, does the agent identify ALL the places that need updating so that TWO reports (with DIFFERENT lookup paths) both come out correct?4 cases · 3 runs · extra task: not in the averageOpen task: rules, cases, every run →