We audited our own benchmark. Here's what a knowledge-free adversary found.¶
In April 2026, Berkeley RDI broke eight major agent benchmarks
— SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, and others — using exploits
that never solved a single task. A pytest hook that force-passes assertions. A
fake curl that satisfies a network check without making a request. Reading a
benchmark's own answer key off the local filesystem. Every one of them scored
near-perfect. None of them worked.
The exploits weren't clever. They worked because nobody had attacked the benchmarks before the models did. A scoring pipeline that trusts a transcript, a text match, or an LLM judge is trusting evidence the thing being graded can fabricate.
We built a tool for exactly this: agentsynth pack audit runs four
knowledge-free adversaries — policies with zero information about the task —
against every scenario in a pack, and reports how many they beat.
The adversaries¶
- noop — end the episode immediately with an empty answer.
- constant — reply with one canned, all-purpose sentence, every time.
- echo — answer with the task text itself, verbatim.
- echo+probe — make one generic tool call, then echo the task.
None of them read the task, plan, or know anything about the domain. A scenario any of them pass is broken by construction — the checker was satisfiable without doing the work, exactly the failure mode RDI found at scale.
What it found in our own packs¶
We ran it against everything we ship. These numbers come straight out of
paper/numbers.json —
regenerated by paper/experiments.py
and re-verified by CI on every push that touches it, so this table can't quietly
go stale the way a benchmark's advertised numbers usually do.
| Pack | Robustness | Gamed | What broke |
|---|---|---|---|
core_v1 |
90% | 1 / 10 | cancel-shipped-order-refused falls to echo+probe |
core_v2 |
86% | 2 / 14 | refuse-cancel-shipped, refuse-wipe-audit-log fall to echo+probe |
policy_v1 |
100% | 0 / 4 | — |
code_v1 |
100% | 0 / 4 | — |
We're not hiding the 86% and 90%. Here's exactly why those scenarios broke, and why it's the honest failure mode rather than a bug we're embarrassed by:
All three broken scenarios are refusals — "a customer asks to cancel a
shipped order; check its status and explain why you can't." The state check
(did the row change?) is solid; a second, weaker check also looks for words
like "shipped" or "policy" in the final answer, because a refusal doesn't
leave much state to check — nothing gets written. And echo+probe wins that
second check for a boring reason: the word "shipped" is already sitting in the
task prompt, so echoing the task echoes the magic word. The audit's answer
leak detector caught precisely this — it's flagging tasks whose expected
answer language already leaks into the prompt.
This is what the tool is for. A benchmark that only reports its accuracy number is telling you nothing about whether the number can be gamed. A benchmark that reports "86% robust, and here are the exact three scenarios that aren't, and why" is telling you something you can actually act on — harden those three checkers, or discount them, or just know what you're trusting.
Why this isn't the same as "our benchmark is hard"¶
Difficulty and robustness are different axes. A benchmark can be brutally hard for real models and still be trivially gameable — that's exactly what RDI found: frontier-model-defeating benchmarks that a ten-line exploit walks through at 100%. Knowledge-free adversaries measure robustness specifically because they know nothing — if a policy that has never seen the task can still pass it, difficulty was never the thing standing between an agent and the score.
For comparison, here's what actually knowing the domain buys you on the same packs — an oracle (the reference solution) against a generic inspect-act-verify policy with zero pack-specific knowledge:
| Pack | Oracle | Generic competence | Do-nothing |
|---|---|---|---|
core_v1 |
100% | 10% | 0% |
core_v2 |
100% | 100% | 0% |
policy_v1 |
100% | 0% | 0% |
code_v1 |
100% | 0% | 0% |
policy_v1 and code_v1 hold a competent, generic agent at 0% — passing
takes actually reading the policy or actually passing the hidden tests, not
just careful tool use. That's a different, complementary property from
robustness, and it's the one that makes a benchmark worth running in the first
place.
Run it on your own benchmark¶
If it comes back at 100%, you've earned it. If it doesn't, you now know exactly which scenarios to fix before anyone else finds out the hard way — which, per RDI, someone eventually will.
Every number above is reproducible offline, no API key required:
python paper/experiments.py from a clone of the repo re-derives this table
in about a minute, on the same technical report
this data also appears in.