Can your agent find the right fact in a long doc?
ranked by score ↓Source
Paste as source: in your trap.yaml
git+https://github.com/trapstreet/trapstreet-tasks@00d0632172c69e6f31c9ce26799ea34865e67930#subdirectory=tasks/core_needle_in_haystackShare
core-needle-in-haystack
An open-source evaluation task for fact retrieval from context — in the shapes AI agents actually encounter: RAG returns, tool outputs, and long documents.
What this task tests
Given a document (RAG chunks or a coherent long doc) containing one or more 4-digit "magic codewords," the model must return the codeword for the specific key named in the question.
Three failure modes covered across the tiers:
- Missed scan — model didn't read the chunk carrying the answer
- Wrong-key grab — model saw multiple similar-format codewords and returned the first/most-salient instead of matching the question's key
- Precision under length — long doc + multiple similar clauses; model can't triangulate the right one
Case structure: 3 tiers × 5 cases = 15 cases
| Tier | Cases | ~Tokens | Scenario | Failure mode |
|---|---|---|---|---|
| 1. retrieval_clean | 5 | ~3k | RAG return: 12 unrelated chunks + 1 needle | Missed scan |
| 2. retrieval_topic_distractors | 5 | ~5k | RAG return: all chunks on same domain, 5 codewords for 5 keys, question asks 1 | Wrong-key grab |
| 3. retrieval_long_doc_precision | 5 | ~24-33k | Coherent long doc (ToS / handbook / API ref / policy / manual) with 5 similar clauses, question asks for 1 | Precision under length |
Each tier maps to a context shape agents commonly receive:
- Tier 1 = short RAG return / small tool output
- Tier 2 = medium RAG return / typical tool output
- Tier 3 = long doc pull (contract, handbook, spec, policy) uploaded by a user
Needle format
Every needle is: "The magic codeword for {key} is {4-digit number}."
For Tier 2 and 3, multiple codewords are planted — one per key in the same domain (e.g., for API tiers: Free, Pro, Enterprise, Legacy, Trial each get a codeword). The question specifies one key. The model must precisely match.
Tier 3 topics
Each Tier 3 case is a full coherent document:
| Case | Document type | 5 keys | Target key |
|---|---|---|---|
service_tos | Terms of Service | MaterialBreach / Convenience / Insolvency / RegulatoryViolation / SecurityBreach | Insolvency |
employee_handbook | Employee Handbook | Annual / Sick / Parental / Bereavement / Sabbatical | Sabbatical |
api_reference | API Reference | Read / Write / Analytics / Admin / Streaming | Analytics |
retention_policy | Data Retention Policy | Transaction / Communication / Personal / Diagnostic / Aggregated | Personal |
product_manual | Product Configuration Manual | Network / Storage / Compute / Security / Monitoring | Security |
Input
Per case:
INPUTS["document.txt"]— the haystack (RAG chunks or long doc)INPUTS["question.txt"]— describes the doc + which key to find + output format
Expected output
ONE 4-digit number. No explanation, no quotes, no prefix.
The judge enforces:
leading_numeric(exact match, tolerance=0)no_hedge(rejects "I couldn't find" / etc.)
Each case scores 1.0 / 0.0. Run passes if ≥80% pass.
Cost
Per full run: ~$0.15 on Haiku-class models, ~$0.44 on Sonnet-class, ~$3.40 on Opus-class.
Honest limitations
- English-only. Multi-language would be a v2.
- Synthetic content. All chunks and long docs are synthetically generated policy / tech-doc-style prose. Real customer docs vary in structure, quality, and language.
- 4-digit numeric needle. Some models cheat by regex-extracting
\d{4}patterns and returning the first hit. The distractor tiers punish this exactly — first-hit strategies get wrong codewords in Tier 2 and 3. no_hedgeis regex-based. A model that declines in novel phrasing won't be caught by the current hedge list.
Data source & license
All content is synthetic, generated by a Python builder using hand-authored templates for tech-documentation prose and templated substitution for policy-style filler. Pattern inspired by gkamradt NIAH (MIT). See LICENSE.md.