Browse use cases
Loading…
An original English, synthetic offline decision-planning benchmark for US third-party consumer unsecured-debt operations. This candidate contains 18 development packets and a fixed 72-case test set, six equally weighted families. It is intended to measure differences between systems without copying public benchmark questions or inventing impossible tasks.
| Family | Test cases | Required decisions |
|---|---|---|
| Ledger reconstruction | 12 | Resolve authenticated revisions, duplicate delivery, pending entries and reversal dependencies across accounts |
| Payment authorization | 12 | Reconstruct scoped consent, holds and transaction status; prevent duplicate or unauthorized charging |
| Contact dispatch | 12 | Respect suppression and fictional conservative contact limits, then find an optimal feasible appointment allocation |
| Dispute protection | 12 | Reconstruct partial dispute holds and correctly scoped release evidence, with specialist protective routing |
| Cashflow planning | 12 | Preserve future essential expenses and cash buffers while planning an affordable authorized installment schedule |
| Settlement selection | 12 | Filter eligibility and select a globally feasible portfolio with consumer-benefit objectives |
All scenario facts and operating policies are provided. Synthetic amounts, event histories and plan combinations are authored here. Source documents inform selected domain concepts, not case text or answers. Complexity tiers change the size of interacting state; they are not calibrated difficulty claims.
The benchmark permits independent computation tools; each run must disclose its harness. The initial calibration compares isolated single-case API calls without external tools and with the same public response schema. An agent with tools is a separate configuration, not an interchangeable score.
Per-case field weights are published in each response contract. The headline score is the mean of six family mean scores, with all 72 cases in the denominator. Whole-case success is reported separately. Optimization families accept every feasible optimum under the declared lexicographic objective: feasibility earns 50%, attaining the optimum earns the remaining 50%; infeasible selections score zero. Other families award typed field accuracy, with ledger balance weight 2 and remaining fields weight 1.
Critical business failures, invalid output, missing delivery, and token-budget truncation must remain separate. High field accuracy does not erase a prohibited charge or an essential-expense breach. The provisional task threshold is at least 85%, no detected critical business failure, full measured coverage and no invalid responses; this is not an industry certification. Complete-case success, safety and reliability must be considered alongside the scalar.
The development/test split, cohort, primary comparison, confidence intervals and repetition plan are specified in methodology.md. A candidate can be discriminative while still easy for one model; non-saturation is reported separately and never manufactured by changing weights after observing test scores. This track does not measure speech, empathy, live tool execution, legal compliance across all jurisdictions or actual cash recovery.
Reference fixture success validates the judge; it is not a model benchmark result. Do not publish reference fixture scores as a model's performance.
All 90 packets are newly generated synthetic scenarios, not real borrower information, scraped calls or copied public evaluation questions. The 18 development cases and 72 test cases use separate deterministic seeds. They share generator families and policy templates; this is held-out scenario-instance evaluation, not proof of generalization to unseen institutions or new workflow families. Private source protects reference answers, but synthetic generation alone is not a durable industry moat. An institution-supplied, independently adjudicated blind set remains necessary for external adoption.
Difficulty comes from joining scoped histories, selecting current authoritative evidence, preserving financial invariants, and satisfying interacting feasibility and optimization constraints. Every case is solvable from its packet. All tied optimal portfolios or schedules are accepted. No demographic attribute is a repayment propensity signal. Protected cash buffers and essential spending cannot be traded away for a better business score.
The documented policy deliberately simplifies some operations. For example, contact slots have already passed local-time and jurisdiction checks; the candidate selects a permitted dispatch portfolio. This is not a test of every Reg F exception or state-specific law. Explicit synthetic rules override assumptions about an institution's real systems.
The following primary sources motivate risk categories; the supplied fictional implementation conventions are not attributed to these laws:
No future-effective FCC amendment is assumed. This candidate does not claim exhaustive federal/state coverage or legal certification.
Freeze the exact input set, scoring code, prompts, model cohort and statistical analysis before opening the holdout. Report every test case; do not remove unfavorable cases or select a successful answer after grading. Separate API delivery retries from additional scored attempts. Missing responses remain coverage gaps, while successfully delivered malformed responses are actual format errors.
Use fresh context per case, preserve provider-returned model identity, token usage, generation IDs and immutable answer hashes. Do not infer prices or usage when provider measurements are missing. Finite completion limits and unsupported providers constrain conclusions.
The comparison protocol uses a fixed compact baseline and a preselected reasoning model, case-matched bootstrap intervals, mixed-item frequency and repeated anchors. The numerical acceptance gates are project criteria rather than an industry standard. Five API-accessible models cannot support a calibrated psychometric item-response model or a claim covering every frontier model.
τ-bench informs reliability measurement; ToolSandbox motivates state-dependent challenges; ACL statistical guidance informs paired comparisons; Ladder motivates protecting holdouts from adaptive reuse. Their public questions are not reused here.
External collections specialists and consumer-protection reviewers have not certified the scenario policies. The initial independent code review recomputed synthetic reference results; it is not expert field validation. Further work should add institution-contributed private cases, real execution traces in an isolated environment, simulator validation, systematic policy-combination holdouts, and separate speech and consumer-treatment tracks.