- A1 · tool-call-scoperuns with your agent
Jev gates: should this tool call run?
An operations agent is about to run one more command. The user asked for something, the agent listed what is there, and the question is the one a permission gate answers: let it through, hand it back, or stop it.45%best · 5 runs · 13d ago · Ruqii - A2 · cve-weakness-class
Jev-class: name the weakness class
Part of decision-layer-bench, which measures whether a small typed decision model can stand in for an LLM classification call inside a pipeline.75%best · 8 runs · 15d ago · Ruqii - A3 · dabstep-shared-toolsruns with your agent
💳 Bigger model or better harness? (DABStep 25, same tools🔧)
A data agent is handed one year of card payments for five merchants, the fee schedule that prices them, and the manual that explains how the schedule is read — and asked 25 questions about them. Five are put straight to the transactions: a count, a fraud rate. The other twenty ta84%best · 6 runs · 16d ago · Ruqii - A4 · model-routingruns with your agent
⚡ Jev as a router: cheaper than always-Opus?
200 multiple-choice questions from MMLU-Pro — 100 law, 100 engineering — where cheap and frontier models are far apart. A solution answers each question however it likes: one model, or a pipeline that sends some questions to a cheaper model and some to a stronger one. The board p91%best · 6 runs · 19d ago · Ruqii - B1 · dabstepruns with your agent
💳 Bigger model or better harness? (DABStep 25)
A data agent is handed one year of card payments for five merchants, the fee schedule that prices them, and the manual that explains how the schedule is read — and asked 25 questions about them. Five are put straight to the transactions: a count, a fraud rate. The other twenty ta84%best · 8 runs · 22d ago · Ruqii - B2 · pdf-chart-reasoningruns with your agent
📊 🧐 Which pdf parser does the best job? read pdf chart
74%best · 7 runs · 1mo ago · Ruqii - B3 · secops-es-investigation
SecOps investigation in Elasticsearch
One alert, one question, a live read-only SIEM — 54 auto-graded SOC investigations over 239k ECS documents in Elasticsearch.86%best · 3 runs · 1mo ago · Zhuaiz - B4 · minecraft-one-liferuns with your agent
💎 LLM Plays Minecraft, One Life 💀
Climb the Minecraft tech tree in a live survival world. Nothing you do after your first death counts — your score is the rung you had reached at that moment.100%best · 1 run · 1mo ago · Ruqii - C1 · session-memory-recallruns with your agent
🧠 Does your memory plugin actually remember?
A solution is given a small ledger and asked to compute one value and remember it. Then, in a separate session, it is asked for that value. The table is gone. Either the value survived the session boundary or it did not.0%best · 1 run · 1mo ago · Ruqii - C2 · ledger-closeruns with your agent
📚 Dsh - A year of receivables. One number. Can your agent get it?
A solution is handed a year of accounts-receivable bookkeeping — monthly ledger extracts, an allocation memo, a customer master, a document index — and asked one figure about the year end. Getting it right means reading the right files, working out an unstated allocation policy f60%best · 4 runs · 1mo ago · Ruqii - C3 · core-capability-stacking-regressionruns with your agent
😬😬 Does adding skills break the jobs an agent already did?
When an agent has more skills installed, does it still do the jobs it already did correctly? Same job, same number of new skills — the only difference is whether they overlap with what the job needs.87%best · 1 run · 1mo ago · Ruqii - C4 · do-llms-dream-of-intjruns with your agent
🐑 Do LLMs Dream of INTJ? 🔮
A trap-compatible task that asks each model to take a 32-question Likert MBTI questionnaire from its own point of view. The judge then computes the 4-letter type and per-axis percentages from the model's responses.6profiles · 50 runs · 22d ago · Ruqii - D1 · pdf-mixed-scanruns with your agent
📃 🧐 Which pdf parser does the best job? Could be any kind of weird pdf
100%best · 6 runs · 1mo ago · Ruqii - D2 · python-bugfix-diffruns with your agent
🪲 Which code review skill works the best?
A code-review Claude Skill (SKILL.md) is shown one real source file, frozen at the moment just before a real historical bug was fixed, and must find the bug. Ground truth is the actual fix commit — not a synthetic injected bug.80%best · 9 runs · 2mo ago · Ruqii - D3 · debug-vendor-payout-pipelineruns with your agent
Debugging in a vendor payout senario - not many cases, but hard 😬
An open-source evaluation task for cross-file consistency debugging — when a ticket asks for a change to a data pipeline, does the agent identify ALL the places that need updating so that TWO reports (with DIFFERENT lookup paths) both come out correct?100%best · 3 runs · 2mo ago · Ruqii - D4 · influencer-marketing-disclosureruns with your agent
influencer_marketing_disclosure
Given a self-contained influencer/creator-partnership scenario, the solution must give correct guidance on the parts of influencer marketing that have actual right answers: whether FTC disclosure is required, how to set up attribution when links aren't clickable, whether to write89%best · 6 runs · 2mo ago · Ruqii - E1 · karpathys-jagged-questions
Karpathy's jagged questions — the 50 m car wash
A trap task built around an anecdote Andrej Karpathy told at the Sequoia AI Ascent fireside chat (April 2026): a state-of-the-art model can refactor a 100k-line codebase, yet advises you to walk to the car wash 50 meters away100%best · 55 runs · 1mo ago · xiaotianhan91 - E2 · mineral-species-id
Mineral Identification from Field Observations
Given a field geologist's hand-specimen observations of an unknown mineral — crystal system, Mohs hardness, streak color, body color, luster, and specific gravity — name the single most likely mineral species out of 98 candidates. This task measures how well a model identifies mi92%best · 14 runs · 2mo ago · Zhuaiz - E3 · minecraft-obtain-diamond
LLM Plays REAL Minecraft - Can Your AI Mine a Diamond? 💎
Can your model actually play Minecraft and come out holding a diamond? This is the classic long-horizon agent benchmark: from an empty inventory, your agent must climb the entire tech tree, every step gated by the last: 🪵 punch wood → 🛠️ crafting table + wooden pickaxe → ⛏️ mine cobblestone → stone pickaxe → ⚙️ find & mine iron ore → 🔥 smelt it into an iron ingot → iron pickaxe → 🕳️ dig deep and mine diamond ore 💎100%best · 2 runs · 2mo ago · Ruqii
↺ Clear Search19 of 19 boards