run · run_yzox5chy153uvgzy
vendor-payout-multi-model-gpt-5.6-luna-pro
Ruqii/vendor-payout-multi-model/gpt-5.6-luna-pro@8b1b5e8 ↗
on Debugging in a vendor payout senario - not many cases, but hard 😬
total score
0.750
engine
openai/gpt-5.6-luna-pro
cases
3 passed · 1 failed
latency
81.00s
cost
—
submitted
2026-07-25 22:31:12
Run summary
- score
- 0.750
- passed
- ✓
- total cases
- 4
- passed cases
- 3
- skipped cases
- 0
- pass threshold
- 0.500
- cost total
- null
- latency p95
- 35.80 s
- latency total
- 81.00 s
- latency median
- 15.44 s
score by category
supplier_change_dated
100%
royalty_change_with_b2b
100%
supplier_change_alltime
100%
srp_change_recalc_royalty
0%
extras
n_scored=4n_skipped_no_gold=0Environment
Self-reported by the solution. Not validated.
- os
- macOS 14.0
- arch
- arm64
- kernel
- Darwin 23.0.0
- cpu
- Apple M1 · 8 cores
- memory
- 8.0 GB
- trap
- 0.0.8
Per-case results
3 pass1 fail
reasonvendor_statement differs; agent output first 300 chars: 'supplier_name,transaction_count,total_revenue_usd,total_deductions_usd,total_net_usd,total_payout_usd\nAnchor Distributors,10,2325.00,29.55,2295.45,1820.56\nBayLine Distributors,5,730.00,23.30,706.70,479.69\nHeartland Distributors,4,530.00,23.25,506.75,352.17\n'
- id
- case_03
- category
- srp_change_recalc_royalty
- difficulty
- srp_change_recalc_royalty
- agent_output
- [{"file": "transactions.csv", "op": "update", "match": {"transaction_id": "TXN-234A"}, "set": {"revenue_gross_usd": 100.0, "vendor_payout_usd": 72.38}}, {"file": "transactions.csv", "op": "update", "match": {"transaction_id": "TXN-234B"}, "set": {"revenue_gross_usd": 120.0, "vendor_payout_usd": 87.75}}, {"file": "transactions.csv", "op": "update", "match": {"transaction_id": "TXN-234C"}, "set": {"revenue_gross_usd": 110.0, "vendor_payout_usd": 80.44}}]
- gold_edit_count
- 3
- agent_edit_count
- 3