Repro notes: the jagged boundary of gpt models on the 50 m car wash (5.5 passes / 5.4-tier falls in)

opened by Zhuaiz · 13d ago

Zhuaiz · 13d ago

Ran several gpt models through the same pass-through wiring (question fed verbatim to the API, no prompt tricks). Results:

Model	Result	Answer opens with	Cost
gpt-5.5	✅ 1	"DRIVE — You need to bring the car to the car wash…"	~$0.0035
gpt-5.4-mini	✗ 0	"WALK — …avoids the hassle of moving the car just to wash it."	~$0.0002
gpt-5.4-nano	✗ 0	"WALK — it's only 50 meters, so walking is quicker…"	~$0.00006

5.5 run: https://trapstreet.run/runs/run_wzg8jj6nq3ywqrg1
nano run: https://trapstreet.run/runs/run_80phxqnhrselqghy
Also re-ran @xiaotianhan91's gpt-5.5 / gpt-5.4-mini solutions locally — results match the leaderboard. Reproducible.

Zhuaiz · 13d ago
A fun detail: gpt-5.4-mini literally says "avoids the hassle of moving the car just to wash it" — the concept of bringing the car is right there in its own words, yet it still picks WALK. It didn't miss the car; it just ranked "less hassle" above "the car has to be there".
Zhuaiz · 13d ago
Two wiring differences for anyone reproducing: my runs use Chat Completions + temperature=0 + raw urllib; @xiaotianhan91's use the Responses API + default params + official SDK. Same conclusion on this task either way (5.5 passes, 5.4-tier fails) — these differences didn't affect the outcome.

reply