session-memory-recall · memory · ranked by score ↓
Share
🧠 Does your memory plugin actually remember?
- runs
- 1
- solutions
- 1
- spent
- $0.17
8 cases
Each case feeds files from inputs/<id>/ to the solution, expects files in expected/<id>/, and is scored by judge.py then aggregated by grader.py.
cases (8)
▸case_01Closing balance on the account; recall it in a fresh session.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 7,526.31
2026-01-06 AR-2026-7261 Invoice raised Fairmont Supplies 7,298.72 14,825.03
2026-01-08 AR-2026-7262 Discount allowed Eastgate Packaging 5,971.91 8,853.12
2026-01-11 AR-2026-7263 Payment received Anchor Distributors 4,391.04 4,462.08
2026-01-16 AR-2026-7264 Settlement received Eastgate Packaging 4,462.08 0.00
2026-01-23 AR-2026-7265 Service billed Fairmont Supplies 5,424.78 5,424.78
2026-01-26 AR-2026-7266 Payment received Fairmont Supplies 5,424.78 0.00
2026-02-01 AR-2026-7267 Late-payment charge Cedarworks Ltd 4,564.58 4,564.58
2026-02-08 AR-2026-7268 Goods dispatched Anchor Distributors 6,301.10 10,865.68
2026-02-12 AR-2026-7269 Freight recharged Fairmont Supplies 5,777.81 16,643.49
2026-02-14 AR-2026-7270 Settlement received BayLine Foods 3,503.59 13,139.90
2026-02-21 AR-2026-7271 Service billed Cedarworks Ltd 2,401.18 15,541.08
2026-02-27 AR-2026-7272 Goods dispatched Cedarworks Ltd 3,368.53 18,909.61
2026-03-04 AR-2026-7273 Goods dispatched Cedarworks Ltd 3,649.09 22,558.70
2026-03-09 AR-2026-7274 Freight recharged Eastgate Packaging 4,974.83 27,533.53
2026-03-14 AR-2026-7275 Settlement received Eastgate Packaging 6,643.65 20,889.88
2026-03-18 AR-2026-7276 Invoice raised BayLine Foods 6,546.54 27,436.42
What is the closing balance on this account at the end of the period?
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the closing balance on the account.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_01",
"category": "closing_balance",
"answer": "27436.42",
"answer_kind": "money"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_02Same question over a longer ledger.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 15,622.44
2026-01-03 AR-2026-5719 Service billed BayLine Foods 1,031.24 16,653.68
2026-01-05 AR-2026-5720 Invoice raised Eastgate Packaging 8,729.39 25,383.07
2026-01-09 AR-2026-5721 Payment received Fairmont Supplies 6,462.27 18,920.80
2026-01-14 AR-2026-5722 Payment received Anchor Distributors 1,765.01 17,155.79
2026-01-21 AR-2026-5723 Credit note issued BayLine Foods 2,166.12 14,989.67
2026-01-25 AR-2026-5724 Credit note issued Eastgate Packaging 1,045.39 13,944.28
2026-02-02 AR-2026-5725 Refund applied Delta Freight Co 8,065.95 5,878.33
2026-02-05 AR-2026-5726 Service billed Fairmont Supplies 1,574.53 7,452.86
2026-02-12 AR-2026-5727 Service billed Cedarworks Ltd 5,225.64 12,678.50
2026-02-19 AR-2026-5728 Refund applied Anchor Distributors 8,458.06 4,220.44
2026-02-21 AR-2026-5729 Service billed Fairmont Supplies 7,882.44 12,102.88
2026-02-23 AR-2026-5730 Invoice raised Anchor Distributors 9,132.42 21,235.30
2026-02-26 AR-2026-5731 Refund applied Delta Freight Co 505.32 20,729.98
2026-03-01 AR-2026-5732 Payment received Anchor Distributors 2,191.85 18,538.13
2026-03-07 AR-2026-5733 Invoice raised Delta Freight Co 1,614.67 20,152.80
2026-03-14 AR-2026-5734 Payment received Delta Freight Co 1,192.73 18,960.07
2026-03-17 AR-2026-5735 Invoice raised Fairmont Supplies 7,418.00 26,378.07
2026-03-23 AR-2026-5736 Late-payment charge Delta Freight Co 6,945.94 33,324.01
What is the closing balance on this account at the end of the period?
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the closing balance on the account.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_02",
"category": "closing_balance",
"answer": "33324.01",
"answer_kind": "money"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_03Voucher number of the largest debit entry; recall it in a fresh session.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 22,255.27
2026-01-04 AR-2026-5804 Invoice raised Fairmont Supplies 680.71 22,935.98
2026-01-10 AR-2026-5805 Payment received BayLine Foods 9,078.95 13,857.03
2026-01-17 AR-2026-5806 Invoice raised Anchor Distributors 5,178.46 19,035.49
2026-01-23 AR-2026-5807 Service billed Anchor Distributors 8,141.60 27,177.09
2026-01-27 AR-2026-5808 Invoice raised BayLine Foods 1,230.56 28,407.65
2026-02-03 AR-2026-5809 Payment received Anchor Distributors 4,567.00 23,840.65
2026-02-06 AR-2026-5810 Invoice raised Fairmont Supplies 7,346.67 31,187.32
2026-02-09 AR-2026-5811 Invoice raised Cedarworks Ltd 4,735.85 35,923.17
2026-02-15 AR-2026-5812 Freight recharged Fairmont Supplies 2,970.51 38,893.68
2026-02-17 AR-2026-5813 Payment received Cedarworks Ltd 3,667.25 35,226.43
2026-02-20 AR-2026-5814 Service billed Cedarworks Ltd 866.60 36,093.03
2026-02-22 AR-2026-5815 Invoice raised Cedarworks Ltd 6,823.47 42,916.50
2026-02-25 AR-2026-5816 Payment received Cedarworks Ltd 3,075.38 39,841.12
2026-02-27 AR-2026-5817 Payment received Eastgate Packaging 9,331.95 30,509.17
2026-03-01 AR-2026-5818 Credit note issued Eastgate Packaging 6,764.15 23,745.02
2026-03-04 AR-2026-5819 Goods dispatched Eastgate Packaging 6,737.57 30,482.59
Identify the Voucher number of the single largest debit entry.
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the voucher number of the largest debit entry.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_03",
"category": "largest_debit_voucher",
"answer": "AR-2026-5807",
"answer_kind": "voucher"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_04Same question over a longer ledger.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 22,026.82
2026-01-03 AR-2026-5185 Service billed BayLine Foods 3,893.11 25,919.93
2026-01-10 AR-2026-5186 Invoice raised Delta Freight Co 2,494.24 28,414.17
2026-01-17 AR-2026-5187 Service billed Anchor Distributors 791.75 29,205.92
2026-01-20 AR-2026-5188 Goods dispatched Eastgate Packaging 7,762.05 36,967.97
2026-01-22 AR-2026-5189 Invoice raised Anchor Distributors 2,125.52 39,093.49
2026-01-28 AR-2026-5190 Freight recharged Delta Freight Co 3,223.74 42,317.23
2026-02-01 AR-2026-5191 Settlement received Eastgate Packaging 5,832.31 36,484.92
2026-02-08 AR-2026-5192 Goods dispatched Anchor Distributors 1,898.37 38,383.29
2026-02-14 AR-2026-5193 Invoice raised Eastgate Packaging 1,469.45 39,852.74
2026-02-19 AR-2026-5194 Invoice raised Anchor Distributors 2,472.20 42,324.94
2026-02-25 AR-2026-5195 Service billed BayLine Foods 4,625.23 46,950.17
2026-02-27 AR-2026-5196 Service billed Anchor Distributors 591.38 47,541.55
2026-03-04 AR-2026-5197 Service billed BayLine Foods 1,057.75 48,599.30
2026-03-11 AR-2026-5198 Service billed Cedarworks Ltd 2,924.51 51,523.81
2026-03-17 AR-2026-5199 Refund applied BayLine Foods 6,803.77 44,720.04
2026-03-19 AR-2026-5200 Invoice raised Fairmont Supplies 7,835.70 52,555.74
2026-03-21 AR-2026-5201 Late-payment charge Anchor Distributors 2,320.02 54,875.76
2026-03-27 AR-2026-5202 Invoice raised Fairmont Supplies 3,016.91 57,892.67
2026-04-03 AR-2026-5203 Service billed Fairmont Supplies 2,640.41 60,533.08
2026-04-08 AR-2026-5204 Service billed Anchor Distributors 5,990.42 66,523.50
Identify the Voucher number of the single largest debit entry.
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the voucher number of the largest debit entry.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_04",
"category": "largest_debit_voucher",
"answer": "AR-2026-5200",
"answer_kind": "voucher"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_05Total debits posted in one calendar month; recall the total in a fresh session.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 21,612.05
2026-01-06 AR-2026-7573 Freight recharged Delta Freight Co 7,709.79 29,321.84
2026-01-08 AR-2026-7574 Invoice raised Anchor Distributors 4,173.18 33,495.02
2026-01-10 AR-2026-7575 Invoice raised Cedarworks Ltd 4,555.26 38,050.28
2026-01-12 AR-2026-7576 Goods dispatched Fairmont Supplies 1,754.64 39,804.92
2026-01-17 AR-2026-7577 Invoice raised Anchor Distributors 2,902.92 42,707.84
2026-01-22 AR-2026-7578 Settlement received Delta Freight Co 6,745.66 35,962.18
2026-01-29 AR-2026-7579 Invoice raised Eastgate Packaging 1,224.84 37,187.02
2026-02-02 AR-2026-7580 Credit note issued Anchor Distributors 9,097.93 28,089.09
2026-02-08 AR-2026-7581 Payment received Anchor Distributors 2,591.74 25,497.35
2026-02-10 AR-2026-7582 Goods dispatched BayLine Foods 7,134.02 32,631.37
2026-02-12 AR-2026-7583 Invoice raised Anchor Distributors 7,887.23 40,518.60
2026-02-14 AR-2026-7584 Invoice raised Eastgate Packaging 2,441.09 42,959.69
2026-02-18 AR-2026-7585 Invoice raised Fairmont Supplies 4,231.74 47,191.43
2026-02-25 AR-2026-7586 Invoice raised Eastgate Packaging 8,452.68 55,644.11
2026-03-01 AR-2026-7587 Credit note issued Fairmont Supplies 9,131.80 46,512.31
2026-03-05 AR-2026-7588 Invoice raised Eastgate Packaging 1,183.25 47,695.56
2026-03-11 AR-2026-7589 Service billed Delta Freight Co 4,639.46 52,335.02
2026-03-15 AR-2026-7590 Invoice raised BayLine Foods 8,418.66 60,753.68
2026-03-19 AR-2026-7591 Settlement received Delta Freight Co 3,125.79 57,627.89
2026-03-22 AR-2026-7592 Invoice raised Fairmont Supplies 5,479.63 63,107.52
2026-03-28 AR-2026-7593 Payment received Cedarworks Ltd 6,940.41 56,167.11
2026-04-01 AR-2026-7594 Settlement received Fairmont Supplies 2,234.83 53,932.28
Total the Debit column over every entry dated in March 2026.
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the total debits posted in March 2026.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_05",
"category": "period_debits",
"answer": "19721.00",
"answer_kind": "money"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_06Total billed to one counterparty; recall the total in a fresh session.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 6,996.61
2026-01-04 AR-2026-6375 Invoice raised Fairmont Supplies 4,218.93 11,215.54
2026-01-08 AR-2026-6376 Invoice raised Fairmont Supplies 8,608.16 19,823.70
2026-01-15 AR-2026-6377 Invoice raised Fairmont Supplies 9,057.21 28,880.91
2026-01-19 AR-2026-6378 Goods dispatched Fairmont Supplies 5,248.67 34,129.58
2026-01-26 AR-2026-6379 Freight recharged Cedarworks Ltd 7,814.48 41,944.06
2026-01-29 AR-2026-6380 Payment received Eastgate Packaging 4,293.69 37,650.37
2026-02-04 AR-2026-6381 Goods dispatched Fairmont Supplies 5,429.73 43,080.10
2026-02-11 AR-2026-6382 Refund applied Delta Freight Co 5,417.50 37,662.60
2026-02-15 AR-2026-6383 Payment received Eastgate Packaging 1,471.20 36,191.40
2026-02-20 AR-2026-6384 Payment received Cedarworks Ltd 7,879.29 28,312.11
2026-02-25 AR-2026-6385 Invoice raised Anchor Distributors 4,963.82 33,275.93
2026-02-27 AR-2026-6386 Payment received Eastgate Packaging 6,909.38 26,366.55
2026-03-01 AR-2026-6387 Goods dispatched Cedarworks Ltd 6,126.22 32,492.77
2026-03-04 AR-2026-6388 Invoice raised Eastgate Packaging 6,655.90 39,148.67
2026-03-09 AR-2026-6389 Settlement received Eastgate Packaging 1,098.89 38,049.78
2026-03-13 AR-2026-6390 Goods dispatched Eastgate Packaging 2,301.22 40,351.00
Total the Debit column over every entry whose Counterparty is Fairmont Supplies.
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the total billed to Fairmont Supplies.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_06",
"category": "counterparty_total",
"answer": "32562.70",
"answer_kind": "money"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_07Same question, different month and ledger.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 8,441.35
2026-01-03 AR-2026-7362 Payment received Eastgate Packaging 7,620.79 820.56
2026-01-08 AR-2026-7363 Payment received Cedarworks Ltd 820.56 0.00
2026-01-12 AR-2026-7364 Invoice raised BayLine Foods 6,735.49 6,735.49
2026-01-17 AR-2026-7365 Invoice raised Anchor Distributors 1,785.58 8,521.07
2026-01-23 AR-2026-7366 Payment received Eastgate Packaging 7,388.69 1,132.38
2026-01-26 AR-2026-7367 Payment received Cedarworks Ltd 1,132.38 0.00
2026-02-02 AR-2026-7368 Invoice raised Delta Freight Co 888.36 888.36
2026-02-09 AR-2026-7369 Invoice raised Fairmont Supplies 2,964.11 3,852.47
2026-02-12 AR-2026-7370 Invoice raised Cedarworks Ltd 259.58 4,112.05
2026-02-14 AR-2026-7371 Refund applied BayLine Foods 4,112.05 0.00
2026-02-18 AR-2026-7372 Freight recharged Anchor Distributors 4,214.10 4,214.10
2026-02-22 AR-2026-7373 Payment received BayLine Foods 4,214.10 0.00
2026-02-25 AR-2026-7374 Invoice raised BayLine Foods 365.00 365.00
2026-02-28 AR-2026-7375 Invoice raised Fairmont Supplies 7,305.44 7,670.44
2026-03-03 AR-2026-7376 Invoice raised Cedarworks Ltd 8,003.27 15,673.71
2026-03-07 AR-2026-7377 Invoice raised Cedarworks Ltd 2,286.24 17,959.95
Total the Debit column over every entry dated in February 2026.
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the total debits posted in February 2026.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_07",
"category": "period_debits",
"answer": "15996.59",
"answer_kind": "money"
}Scored by judge.py — see Scoring logic below for the full rule.
▸case_08Voucher number at a given rank among debits; recall it in a fresh session.
input
README.md
# Case protocol
*Addressed to the solution author. Your agent never reads this file -- it
sees `step1.txt` or `step2.txt` as its prompt, one per session.*
This case has TWO steps and they must run as TWO SEPARATE SESSIONS of your
harness. That separation is the entire point of the task.
1. Run `step1.txt` as the prompt for a session. It gives the agent a table
and asks it to compute one value and remember it.
2. Run `step2.txt` as the prompt for a **fresh session**. It asks for that
value. The table is not repeated.
**Use `tools/run_case.sh` and none of the rules below can be violated by
accident.** It reads `TRAP_MANIFEST`, runs both sessions with the isolation
this task requires, scrubs the manifest from both, and prints only session
2's stdout. Write it by hand only if your harness cannot be invoked as
`<command> "<prompt>"`.
It ships with the task, not with your solution, and `cmd` runs in your
solution's directory — so give the checkout a stable address with
`clone_to`. The task README has the four-line recipe.
## Rules
- Step 2 MUST be a new session, not a continuation or a resume of step 1.
Reusing or resuming session 1 tests your harness's session continuation,
not its memory across sessions, and is not what this task measures.
- **Step 2's working directory must not reach step 1's files.** Run the two
steps in unrelated directories, and do not leave `step1.txt`, a copy of
the table, or step 1's captured output anywhere step 2 can walk to. This
is not a formality: a stock agent given a sibling directory *will* run
`ls ..`, find `step1.txt`, and recompute the answer from the table
instead of recalling it. That scores 1.0 and measures nothing.
- Your solution's stdout is what gets scored, and only step 2's answer
belongs there. Print the value on the last non-empty line.
- Nothing forbids you from carrying the value yourself instead of letting
your agent remember it -- but see "Known limitations" in the task README.
Solutions are public.
## Output
The last non-empty line of stdout must be the value from step 2, alone.
If your solution has no value to report, print `UNKNOWN`.
step1.txt
You are working through a two-part job. This is PART 1 of 2.
Below is an extract from an internal ledger.
Accounts Receivable — subsidiary ledger
Account 1200 · Trade Debtors · period 2026-01-01 to 2026-03-31 · USD
Date Voucher Narration Counterparty Debit Credit Balance
----------------------------------------------------------------------------------------------------------
2026-01-01 Balance brought forward 7,238.82
2026-01-06 AR-2026-7551 Service billed Fairmont Supplies 5,533.10 12,771.92
2026-01-13 AR-2026-7552 Service billed Anchor Distributors 5,262.65 18,034.57
2026-01-18 AR-2026-7553 Service billed Cedarworks Ltd 6,835.69 24,870.26
2026-01-21 AR-2026-7554 Invoice raised Cedarworks Ltd 2,514.94 27,385.20
2026-01-26 AR-2026-7555 Credit note issued Eastgate Packaging 8,438.52 18,946.68
2026-02-03 AR-2026-7556 Payment received Anchor Distributors 8,749.86 10,196.82
2026-02-09 AR-2026-7557 Invoice raised Anchor Distributors 6,171.48 16,368.30
2026-02-16 AR-2026-7558 Payment received Delta Freight Co 6,849.63 9,518.67
2026-02-23 AR-2026-7559 Discount allowed Eastgate Packaging 4,067.11 5,451.56
2026-02-25 AR-2026-7560 Payment received Anchor Distributors 5,451.56 0.00
2026-03-01 AR-2026-7561 Invoice raised Fairmont Supplies 8,734.43 8,734.43
2026-03-04 AR-2026-7562 Invoice raised Cedarworks Ltd 7,800.13 16,534.56
2026-03-09 AR-2026-7563 Payment received Delta Freight Co 7,437.93 9,096.63
2026-03-16 AR-2026-7564 Goods dispatched Fairmont Supplies 8,626.11 17,722.74
2026-03-18 AR-2026-7565 Invoice raised Cedarworks Ltd 1,260.87 18,983.61
2026-03-20 AR-2026-7566 Invoice raised Anchor Distributors 3,857.25 22,840.86
2026-03-24 AR-2026-7567 Service billed Eastgate Packaging 2,842.12 25,682.98
Rank the debit entries from largest to smallest (break ties by Voucher, ascending) and identify the Voucher number at position 2.
Do the calculation now, then state the result once, in one short sentence,
and commit it to memory. You will be asked for it later in a SEPARATE
session -- the table below will NOT be available then, so remembering the
result is the point of this step.
step2.txt
This is PART 2 of 2, continuing the job you started earlier.
Earlier you worked through a ledger extract and computed one value:
the voucher number at position 2 when debits are ranked largest first.
Report that value now.
Output ONLY the value on the last line -- no explanation, no units, no
quotes, no surrounding prose. If you genuinely do not have the value,
output exactly: UNKNOWN
expected output
answer.json
{
"id": "case_08",
"category": "nth_largest_voucher",
"answer": "AR-2026-7564",
"answer_kind": "voucher"
}Scored by judge.py — see Scoring logic below for the full rule.
scoring logic
judge.py runs once per case and prints a score per case. grader.py runs once at the end and folds case scores into a run-level summary. Without grader.py, the run's score is simply the average of case scores.
▸judge.py184 lines · view on GitHub
"""Per-case judge for session_memory_recall.
I/O contract: reads TRAPTASK_MANIFEST (trap-cli).
Scoring is binary and fully deterministic (no LLM judge). The answer is a
machine-generated number or record id, never a sentence, so there is no
phrasing to argue about -- either the value came back or it did not.
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
VOUCHER_RE = re.compile(r"\bAR-2026-\d{4}\b")
ISO_DATE_RE = re.compile(r"\b\d{4}-\d{2}-\d{2}\b")
# Money as it may plausibly be written back: 12345.67, 12,345.67, $12,345.67.
# The trailing lookahead rejects a following digit, and a following period only
# when a digit follows it -- a sentence-ending period is not part of the number.
# Getting this wrong is not cosmetic: `(?![\w.])` made "27,940.01." match as
# "27" and "27940.01." match as nothing at all, so any value that ended a
# sentence was invisible here.
MONEY_RE = re.compile(
r"(?<![\w.])\$?\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?(?![\w])(?!\.\d)"
r"|(?<![\w.])\$?\d+(?:\.\d{1,2})?(?![\w])(?!\.\d)"
)
STRIP = " \t\r\n\"'`*.,:;!?()[]{}"
# Mirrors assert_answer_hard_to_guess() in build_cases.py: every money answer
# carries at least four digits before the decimal point. Anything shorter is
# therefore not the answer -- it is a step number, a match count, a rank -- and
# counting it as a rival candidate only produced false negatives.
MIN_WHOLE_DIGITS = 4
def last_value_line(stdout: str) -> str:
lines = [ln.strip(STRIP) for ln in stdout.splitlines()]
lines = [ln for ln in lines if ln]
return lines[-1] if lines else ""
def normalise_money(text: str) -> str | None:
"""'$12,345.67' -> '12345.67'. None if it is not a money-shaped token."""
t = text.replace("$", "").replace(",", "").replace("USD", "").strip()
try:
return f"{float(t):.2f}"
except ValueError:
return None
def candidates(text: str, kind: str) -> set[str]:
"""Every token in `text` that could plausibly be an answer of this kind.
Used to reject shotgun answers -- printing every figure in the ledger must
not count as remembering the one that mattered. Which makes the set's
precision matter in both directions: a token that could never be the answer
must not land here, or it turns a correct answer into a rival candidate and
the shotgun rule fires on a solution that did nothing wrong. Voucher ids
and ISO dates are removed before the money scan for exactly that reason --
"AR-2026-0016" otherwise contributed 2026.00 and 16.00.
"""
if kind == "voucher":
return set(VOUCHER_RE.findall(text))
scrubbed = ISO_DATE_RE.sub(" ", VOUCHER_RE.sub(" ", text))
out = set()
for tok in MONEY_RE.findall(scrubbed):
n = normalise_money(tok)
if n is None or len(n.split(".")[0]) < MIN_WHOLE_DIGITS:
continue
out.add(n)
return out
def score_case(stdout: str, expected: dict) -> dict[str, Any]:
"""1.0 iff the remembered value came back.
The final line is the contract, so it is read first and it is decisive:
if it carries a value of the right kind, that value is the answer the
solution is giving, whether or not prose surrounds it.
1. the final line IS the value -> 1.0
2. the final line carries exactly one candidate -> that candidate
3. the final line carries several -> 0.0, ambiguous
4. the final line carries none -> fall back to the
whole output, crediting it only if the answer is the sole candidate
Step 4 forgives a solution that trails off after answering. Steps 2-3 are
what stop a shotgun: listing every figure in the ledger is not remembering
the one that mattered.
"""
answer = expected["answer"]
kind = expected.get("answer_kind", "money")
category = expected.get("category")
tail = last_value_line(stdout)
tail_norm = tail if kind == "voucher" else (normalise_money(tail) or tail)
if tail.upper() == "UNKNOWN":
return {
"score": 0.0,
"reason": "solution reported UNKNOWN -- value did not survive into session 2",
"category": category,
"reported": "UNKNOWN",
}
if tail_norm.lower() == answer.lower():
return {"score": 1.0, "reason": "exact match on final line", "category": category,
"reported": tail}
tail_found = candidates(tail, kind)
if tail_found:
if answer in tail_found and len(tail_found) == 1:
return {
"score": 1.0,
"reason": "value on the final line (surrounding prose tolerated)",
"category": category,
"reported": answer,
}
if answer in tail_found:
return {
"score": 0.0,
"reason": f"final line carries {len(tail_found)} candidates -- shotgun not credited",
"category": category,
"reported": tail,
"n_candidates": len(tail_found),
}
return {
"score": 0.0,
"reason": "final line reports a different value",
"category": category,
"reported": tail,
"n_candidates": len(tail_found),
}
found = candidates(stdout, kind)
if answer in found and len(found) == 1:
return {
"score": 1.0,
"reason": "value present and unambiguous (preamble tolerated)",
"category": category,
"reported": answer,
}
if answer in found:
return {
"score": 0.0,
"reason": f"value present but {len(found)} candidates in output -- shotgun not credited",
"category": category,
"reported": tail,
"n_candidates": len(found),
}
return {
"score": 0.0,
"reason": "value absent from output",
"category": category,
"reported": tail,
"n_candidates": len(found),
}
def main() -> None:
m = json.loads(os.environ["TRAPTASK_MANIFEST"])
stdout = Path(m["run"]["stdout"]).read_text(errors="replace")
exit_code = json.loads(Path(m["run"]["meta"]).read_text())["exit_code"]
expected = json.loads((Path(m["expected_dir"]) / "answer.json").read_text())
if exit_code != 0:
result = {"score": 0.0, "reason": f"solution exited {exit_code}",
"category": expected.get("category")}
elif not stdout.strip():
result = {"score": 0.0, "reason": "empty stdout",
"category": expected.get("category")}
else:
result = score_case(stdout, expected)
result["id"] = expected["id"]
print(json.dumps(result))
if __name__ == "__main__":
main()
▸grader.py79 lines · view on GitHub
"""Overall grader for session_memory_recall.
Aggregates per-case judge results (the trap-cli TRAPTASK_MANIFEST list)
into a run-level verdict. This aggregation logic is standard across every
task in this repo -- usually nothing to customize here. If your judge's
metrics dict uses a different field name than "bug_category" for its
category breakdown, update CATEGORY_FIELD below; otherwise leave this file
as-is.
"""
from __future__ import annotations
import json
import os
from collections import Counter
PASS_THRESHOLD = 0.5
CATEGORY_FIELD = "category" # change to match your judge.py's metrics dict, or None to disable
def main() -> None:
cases = json.loads(os.environ["TRAPTASK_MANIFEST"])
scored = [c for c in cases if c.get("metrics") and c["metrics"].get("score") is not None]
skipped = [c for c in cases if not c.get("metrics") or c["metrics"].get("score") is None]
accuracy = sum(c["metrics"]["score"] for c in scored) / len(scored) if scored else 0.0
by_category_pct = {}
if CATEGORY_FIELD:
by_category_score: Counter[str] = Counter()
by_category_total: Counter[str] = Counter()
for c in scored:
cat = c["metrics"].get(CATEGORY_FIELD)
if cat:
by_category_total[cat] += 1
by_category_score[cat] += c["metrics"]["score"]
by_category_pct = {
k: round(by_category_score[k] / by_category_total[k], 3) for k in by_category_total
}
durations = [c.get("duration", 0.0) for c in cases if c.get("duration") is not None]
if durations:
ds = sorted(durations)
latency_ms_median = round(ds[len(ds) // 2] * 1000, 1)
latency_ms_p95 = round(ds[int(0.95 * len(ds))] * 1000, 1) if len(ds) > 1 else latency_ms_median
latency_ms_total = round(sum(ds) * 1000, 1)
else:
latency_ms_median = latency_ms_p95 = latency_ms_total = 0.0
case_costs = [
c["cost"]["cost_usd"]
for c in cases
if isinstance(c.get("cost"), dict) and c["cost"].get("cost_usd") is not None
]
cost_usd_total = round(sum(case_costs), 4) if case_costs else None
n_passed = sum(1 for c in scored if c["metrics"]["score"] == 1.0)
passed = bool(scored) and accuracy >= PASS_THRESHOLD
print(json.dumps({
"passed": passed,
"score": round(accuracy, 3),
"n_passed": n_passed,
"n_total": len(cases),
"n_scored": len(scored),
"n_skipped_no_gold": len(skipped),
"threshold": PASS_THRESHOLD,
"by_category": by_category_pct,
"latency_ms_median": latency_ms_median,
"latency_ms_p95": latency_ms_p95,
"latency_ms_total": latency_ms_total,
"cost_usd_total": cost_usd_total,
}))
if __name__ == "__main__":
main()