run · run_x0ah1zo1jwdnps8v

docling-claude

Ruqii/trapstreet-solutions/docling-claude@80cb002

on Legal Contract Review

total score

0.632

engine

docling + claude-opus-4-7

cases

12 passed · 7 failed

latency

147.05s

cost

$1.695

submitted

2026-05-23 23:55:10

Run summary

score
0.632
passed
total cases
19
passed cases
12
skipped cases
0
pass threshold
0.800
tokens total
null
cost total
$1.695
latency p95
13.20 s
latency total
147.05 s
latency median
7.12 s

score by category

dates
0%
money
29%
clauses
100%
deposit
100%
scenario
0%
scenario_reasoning
100%

extras

n_scored=19n_skipped_no_gold=0

Per-case results

12 pass7 fail

answer

The deposit amount is not legible in the extracted text.

expected

2250
id
deposit_amount
type
numeric
model
claude-opus-4-7
category
money
usd_cost
$0.086
difficulty
easy
input_tokens
30
output_tokens
21
matcher_results
[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0

answer

The 1st of the month.

expected

on or prior 5th of the month
id
rent_payment_day
type
text
model
claude-opus-4-7
category
money
usd_cost
$0.086
difficulty
medium
input_tokens
30
output_tokens
12
matcher_results
[object Object],[object Object],[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0

answer

The document states rent increases but the actual figures appear as garbled/encoded text in the extraction and cannot be read as a clear GBP amount.

expected

2100
id
rent_year2
type
numeric
model
claude-opus-4-7
category
money
usd_cost
$0.089
difficulty
medium
input_tokens
50
output_tokens
49
matcher_results
[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0

answer

N/A

expected

2400
id
rent_year3
type
numeric
model
claude-opus-4-7
category
money
usd_cost
$0.086
difficulty
medium
input_tokens
75
output_tokens
7
matcher_results
[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0

answer

Months remaining at surrender: 36 − 22 = 14 months. Gap = 1 month, so replacement covers 13 months. Rent schedule per the agreement: - Months 1–12: original rent (call it R1) - Months 13–24: increased rent R2 - Months 25–36: further increased rent R3 The tenant is 22 months in, so months 23–36 remain (14 months). - Month 23–24: at R2 (the second-tier rent) - Months 25–36: at R3 (the third-tier rent) The document's rent figures (clause 8a/8b) are not legible in the extract, so I'll express the answer in terms of the rents: (a) Gap rent (1 month, month 23) = R2 (b) Replacement covers 13 months: month 24 at R2, months 25–36 (12 months) at R3. Letting fee = 13.2% × (1×R2 + 12×R3) (c) Inventory check-in = £144 × 14/36 = £56.00 (d) Administration = £480 × 14/36 = £186.67 **Total = R2 + 0.132 × (R2 + 12·R3) + £242.67** A specific GBP total cannot be computed because the rent amounts in clauses 8(a) and 8(b) are not readable in the provided markdown.

expected

£6,421.47
id
scenario_leave_22mo_replacement_1mo_gap
type
numeric
model
claude-opus-4-7
category
scenario
usd_cost
$0.121
difficulty
hard
input_tokens
275
output_tokens
435
matcher_results
[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0

answer

16/09/2019

expected

05/09/2022
id
term_start
type
date
model
claude-opus-4-7
category
dates
usd_cost
$0.086
difficulty
easy
input_tokens
41
output_tokens
11
matcher_results
[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0

answer

Based on the document, the rent figures (£ amounts) in clauses 8(a) and 8(b) are not legible in the OCR output — the actual numeric values for the monthly rent and the increased rents are not present in the extracted markdown, so the total cannot be computed.

expected

77400
id
total_rent_fixed_term
type
numeric
model
claude-opus-4-7
category
money
usd_cost
$0.092
difficulty
hard
input_tokens
68
output_tokens
90
matcher_results
[object Object]
cache_read_input_tokens
56,211
cache_creation_input_tokens
0
discuss this task →