Blog ·
The code test in the full benchmark checks the basics: short tasks, one answer each. The Hard coding test asks what comes next: when the work gets long and messy, where does your agent stop keeping up?
It has 18 challenges in three tiers and a three-hour budget, shared across all of them. How the agent spends that time is part of the test. Every challenge is generated from a random seed when the run starts: new data, new bugs, new names, a new CPU, a new city. The agent works on its own machine with its own tools. We never run its code: it sends back a text answer, a checksum or a list of results, and we compare it with what our generator computed.
Every challenge is pass or fail, and the score is simply how many passed. When a task deserves partial credit, it is split into several challenges instead. That is what v2 changed: v1 scored the CPU out of 6 and the routes in thirds, and a report had no clear way to show those fractions. Scores from the two versions can’t be compared.
Three short tasks a capable agent finishes in a few minutes each. They are allowed to be easy. They check that the basics work before the rest of the test asks for more.
A small Python project with one failing unit test, caused by one planted bug. The agent downloads it, finds the wrong line, fixes it, runs the program and reports the 8-character checksum it prints.
The project is new on every run: the bug is drawn from a catalog, and every function, module and domain noun is renamed from a pool of synonyms. Names never reach the output, so the checksum depends only on the code being right.
Say exactly what a JavaScript program prints. It is stitched from five small snippets, each built around a classic trap: switch fall-through, closures in a loop, and the like.
Running the program is allowed. The test is whether the agent reads code precisely, or checks it, rather than guessing what it probably does.
A short written spec, in any language the agent likes, then 15 listed inputs to run it on. Every one of the 15 result lines has to match. The edge cases are in the spec, and they are in the inputs too.
Six tasks shaped like SWE-bench work: a crash report, a feature request, a program that breaks its own README, a bug far from its symptom, bugs that hide each other, and code that is correct but far too slow.
We copy the shapes, not the repositories. Real SWE-bench repositories need Docker, are far too big to hand over as a download, and sit in every model's training data. So each run generates a small Python project from a working reference, plants bugs from a catalog, and checks that each bug really changes the output. Python 3.10 or newer, standard library only.
A small orders pipeline: parse, normalize, dedupe, price, tax, allocate, report. A user reports a crash and pastes the traceback. The agent fixes it and reports the checksum for the real data file.
The traceback shows where it crashed, and the README says what should have happened. Skipping the rows that crash is the tempting fix, and it is usually the wrong one.
Same kind of project, but this time a user asks for a new option. The agent adds it without breaking anything else the README promises, then runs the program with the new option.
The checksum in the README describes the old program, so it can't confirm the answer. The agent has to trust its reading of the request.
Every test passes. But the README promises a checksum for the sample data, and the program prints a different one. No traceback, no failing test: the README is the only evidence that something is wrong.
One group's total in the report disagrees with Finance's ledger. The sample data still checks out, so the bug only shows on the real export, and it lives somewhere else than where the wrong number appears.
The agent has to follow one wrong figure back through the pipeline to its real source, with no order to start from and no sample checksum to confirm the fix.
Three group totals disagree with the ledger, and several bugs are involved. Fixing one can hide another or change how it shows. The tests pass and the sample checks out from the start, so only the ledger figures say what is still wrong.
This is the challenge that catches agents who stop after the first fix.
A correct Python log analyzer, and a generated log of a million lines it is far too slow to finish. The agent finds the real slow spots, rewrites them, and reports the checksum for the big log.
The README gives a checksum for a small sample, which proves the output didn't change along the way.
The first draft of this test had nine challenges, ending with the deep debugging above. A local 27B coding model on a single RTX 5090 scored 9 out of 9, and 9 out of 9 again after the hardest ones were made harder. Finding bugs in a small, documented project had hit its ceiling.
So before writing a new tier 3, we tried four candidates on that local model and on a frontier agent. Writing a CPU emulator from a full spec: both passed. An interpreter for a new language from a full spec: both passed. The two that separated them had nothing to do with how hard the code is. One hands the agent far more data than it can read. The other has no single right answer, only better and worse ones.
Those two make up tier 3. Each is split into several challenges that pass or fail on their own, so tier 3 is 9 of the 18 challenges: half the test.
The agent gets a partial spec of a 16-bit CPU: eight registers, four flag names, 64 KiB of memory, the program format, and that is about it. No opcode table, no instruction formats, no flag rules, no word on traps. Then eight probe programs with a full execution trace each, about 11 MB in all, and six new programs with no trace. It must report what each of the six prints and what it leaves in R0.
11 MB of trace doesn't fit in any context window. In our trial run, the frontier agent wrote tools to mine the traces and got all six in nine minutes. The local model tried to read them and ran out of context after 25 minutes, with nothing.
Each target leans on a different corner of the CPU: carries and borrows, bit fields and rotates, which instructions keep which flags, nested traps, and a hash instruction. Missing one corner costs one target, not all six. A target passes only if both the output and the final R0 match.
One day of deliveries for 600 to 800 customers: time windows, vehicle capacity, a fixed cost per vehicle, orders that must ride together, and a downtown zone with a per-minute toll that changes by the hour. The answer is the plan as JSON.
Anyone can produce a valid plan. A greedy one is far from good, and closing the gap takes a real local search. The toll is there so that an off-the-shelf solver that can't see time-dependent costs falls at least one level short.
The plan is checked against every rule and costed exactly. Bronze, silver and gold are the same problem with three cost limits: within 18%, 9% and 4% of an estimate of the best cost known for that instance. A plan good enough for gold passes all three.
Tiers 1 and 2 tell you whether your agent can do everyday coding work on its own. Tier 3 tells you how it copes with too much to read and no single right answer. Those are the two places where a local setup and a frontier one still pull apart, and where a better harness, more context or a smarter prompt can make a visible difference.
comments
Sign in to join the discussion
No messages yet.