Building Document Retrieval Benchmarks That Actually Work
2026-08-30 · by Ruqi
Have you run into this situation? You built your own benchmark and your agent passed every single one. However, it still fails in real-world tasks.
Your instinct is to improve the difficulty of your benchmarks. But what actually counts as hard for an agent? It may not be the same as what feels hard.
This article will introduce 7 levers for document-retrieval benchmarks.
01
Ask for something that is not there
Most questions have answers in your benchmark doc. Try to give some that the doc does not answer. If an agent invents an answer by itself, it is doing exactly the thing that will hurt you in production.
Keep one question that looks unanswerable and is not. Without it, an agent that replies "not derivable" to everything scores well on the rest.
Instead of "What is the 2028 projection?" ask "What is the 2029 projection?" — when the table stops at 2028.
IN PRACTICE
In task pdf-chart-reasoning, cases 18 and 19. The correct answer to both is "not derivable". Case 19 asks which participant placed a dot, and the dots are anonymous; case 18 asks for a figure the release never collects.
02
Aggregate instead of looking up
Reading one cell from your benchmark doc is retrieval, but combining several is the actual work in the real world. An agent has to find all of them first, so a miss anywhere breaks the final answer.
Instead of "What was Q2 revenue?" ask "What was total revenue across the four segments?"
Four cells feed one number. Miss any of them and the sum is wrong.
IN PRACTICE
In task pdf-chart-reasoning, case 01 asks how many participants fall in the 4.2–4.3 percent bin — one bar. Case 09 asks how many project growth below 2.2 percent, which means finding every bin under that line and adding them.
03
Rank by size, not by membership
For a benchmark question, asking which rows match a label only tests each row independently. Asking for a ranking instead forces the agent to compare values across rows before it can answer.
Instead of "Which regions are in EMEA?" ask "Which region had the third-highest margin?"
Membership is per-row. Ranking is not answerable until every row has been seen.
IN PRACTICE
In task pdf-mixed-scan, case 09 asks which of the 12 Reserve Bank districts holds the largest share of deposits, and what share that is. You cannot answer it without reading all 12.
04
Count instead of retrieving
Finding one thing from the benchmark doc is a search that stops as soon as it succeeds. But counting means the agent has to visit every candidate and keep a running total without skipping or double-counting.
Instead of "Which fund held Acme stock?" ask "How many funds held Acme stock?"
A search can stop early. A count cannot.
IN PRACTICE
In task pdf-chart-reasoning, case 12 asks how many dots sit at 3.875 percent. Case 13 asks how many rate levels have at least one dot, so you count positions instead of marks, and must not count a level twice.
05
Make the haystack bigger, not the formula longer
This is the one people get backwards. They complicate the arithmetic and leave the document small. Enlarging the search space is a separate lever and it works on its own: keep the same operation, and give the agent far more candidates to sift before it can apply it.
Instead of a four-step calculation over 12 rows, ask for the same average over 152.
The operation does not change. The number of places its inputs could hide does.
06
Split the evidence across pages
Put one half of the answer on page 3 and the other on page 30. The model has to find both and know they belong together — and nothing on either page says so.
A rate in a table on one page, and the footnote twenty pages later that redefines what it counts.
Neither page tells the model the other one exists.
IN PRACTICE
In task pdf-chart-reasoning, case 22 asks how far apart the June 2020 readings are in figures 4.D and 4.E — one for uncertainty, one for risk, pointing opposite ways. Neither figure mentions the other.
07
Make it go through OCR
If the page arrives as an image, everything downstream inherits the OCR's mistakes. The question can be trivial and still fail, because the text the model reasons over is not quite the text on the page.
A scanned invoice rather than the digital original — same fields, same question.
The model never sees the page. It sees whatever the OCR made of it.
IN PRACTICE
Task pdf-mixed-scan vs task pdf-chart-reasoning. The same OCR arm scores 0.900 on the first, built from scanned statements, and 0.130 on the second, built from rasterised charts — exactly what the plain text-layer arm scores there.
6 of these change the question. Only the last changes what the document is.
But rephrasing your questions is not the only way to raise difficulty. Apply all 7 levers to a document that is easy to read, and your agent will still score 100%.
I will share more tips and experience on building a valid benchmark of your own.
Effects are reported figures from MMLongBench-Doc (NeurIPS 2024), FinSheet-Bench (arXiv 2026) and OHR-Bench (ICCV 2025), each checked against the paper.