An open-source evaluation task for cross-file, cross-report consistency debugging — when a ticket asks for a change to a SaaS billing pipeline, does the agent identify ALL the places that need updating so that FOUR reports (each resolving the same underlying facts through a d
An open-source evaluation task for cross-file consistency debugging — when a ticket asks for a change to a data pipeline, does the agent identify ALL the places that need updating so that TWO reports (with DIFFERENT lookup paths) both come out correct?
A code-review Claude Skill (SKILL.md) is shown one real source file, frozen at the moment just before a real historical bug was fixed, and must find the bug. Ground truth is the actual fix commit — not a synthetic injected bug.