The circulating claim is that adding skills degrades 30β50% of previously
working tasks. core_tool_selection_at_scale tested the obvious version of it β
catalog size, up to 300 tools β and it did not reproduce. So this task holds
count fixed and manipulates the variable the claim's proponents actually
describe: interference between skills that functionally overlap, as distinct
from skills that merely coexist.
Every case exists twice. Same scenario, same required calls, same number of skills added β the arms differ only in whether the added skills compete with the ones the job needs. The low-overlap arm is the control, and it is what makes a drop attributable to overlap rather than to catalog size. Without it, a drop is just as easily explained by the catalog having grown.
12 scenarios Γ 9 variants = 108 cases. L0 is the shared baseline: nothing has been added yet, so both arms are identical there and share one case.
| dimension | levels | what it varies |
|---|---|---|
stack_level | L0 / L1 / L2 / L3 | overlap dose β 8 / 14 / 20 / 26 skills, count equal across arms |
stack_level | L4 | the same contrast at 126 skills (~24k tokens); separately registered, excluded from the curve and the primary test |
overlap_class | high / low | whether the added skills compete with what the job needs |
difficulty | easy / medium / hard / edge | which scenarios enter the primary test β 9 primary (medium/hard), 1 easy canary, 2 edge |
Three invariants make a result attributable, all asserted in tests/ β but read
the control-arm limitation below before treating a measured gap as pure overlap:
disqualifier field records the inference a
reader must make, not the sentence that would give it away.inputs/<case_id>/prompt.txt carries the skill catalog as a JSON schema list,
the user request, and the output instruction. The solution prints one JSON array
to stdout:
[{"name": "<tool_name>", "arguments": {"<arg>": <value>}}]
Order is not scored. Case IDs are opaque on purpose β a solution can read its
own inputs_dir path, so an ID like s04_high_L3 would hand it the condition
it is in.
Deterministic, no LLM-as-judge. A scenario needs a set of calls:
completion = recall β how many required calls arrivedcorrectness = precision β how many emitted calls were rightscore = F1A stacked catalog can break a workflow by dropping a step (recall falls) or by substituting a plausible neighbour (precision falls); those need different fixes, so they are reported separately.
A call matches iff the tool name is exact and every expected argument is
present with an accepted value. Argument matching is deliberately generous,
reused verbatim from core_tool_selection_at_scale.
Emitting every plausible skill collapses precision by construction, so there is no cap on answer length, no positional rule and no required phrasing.
Each shortfall is classified: near_miss, wrong_backend, instruction_bleed,
unsolicited_addition, over_eager, unrelated_tool, bad_arguments,
incomplete, unparseable, solution_error.
score is a compressed proxy, not the measurement. Roughly 43% of the cases
are the control arm β designed to stay flat β and another ~12% is the shared L0
baseline, so only about 43% carries the manipulation. It moves in the right
direction: a solution that fully removed the overlap penalty would lift score
by about 0.065. But it moves at roughly 43% of the amplitude of the thing it is
tracking, and two runs can land within 0.006 of each other on score while
sitting 0.023 apart on the high-overlap arm β which is what happened in
RESULTS.md. Read the rank as a rough ordering and the fields below as the
result.
Four flat scalars carry the actual outcome, so they show up on a run page without digging:
| field | what it is |
|---|---|
high_overlap_score | mean under competing skills β the condition being tested |
low_overlap_score | mean under the control arm |
arm_gap | low β high; an upper bound on the overlap penalty, see limitations |
primary_p | sign-flip permutation on per-scenario mean(low β high) over L1βL3, medium/hard tiers only |
The same values live in by_overlap_class and primary_test, which stay
authoritative for anything reading this programmatically. grader.py also
reports curve_high_overlap + curve_high_drops (the dose response, with the
low arm as control), by_failure_reason, bled_by_instruction_strength,
by_competitor_dose and by_skill_pair.
n_solution_error, score_excluding_solution_errors and
solution_errors_by_stack_level sit beside the headline because provider errors
and context overflow concentrate at L4-high β exactly where the hypothesis
predicts degradation.
RESULTS.md β two full matrix runs and what they establish. CALIBRATION.md β
how the instrument got here, including the v1 that could not fail, the position
confound, and the dead dose plot. The pre-registered analysis is fixed in
grader.py (PRIMARY_TIERS, INFLECTION_MIN_DROP, INFLECTION_RATIO), not
merely in prose.
100% synthetic and hand-authored β see LICENSE.md. Nothing is drawn from an
external corpus. The narrower caveat is ours: an earlier draft reused wording
from core_tool_selection_at_scale, whose README is public. Those were
rewritten, but the two catalogs remain thematically adjacent.
python3 gen_filler.py # regenerate the L4 bulk pool (committed; rarely needed)
python3 gen_grid.py # regenerate the grid + traptask.yaml (committed)
python3 build_cases.py # compose inputs/ + expected/
python3 -m pytest tests/ -v
inputs/ and expected/ are GENERATED. Edit catalog.json (the skills) or
scenarios.json (the jobs) and rebuild β never edit them by hand.