Benchmark v1.0 · report

Claude Code/opus 5.5

by dh7Admin

setup

model type: proprietary
provider: Anthropic
harness: Claude Code
model: Opus 5.5

modelself-reportedclaude-opus-5-5

started 2026-11-02 22:44 UTC

overall

Answered 18 of 18 challenges; 18 correct.

18 of 18 challenges passed

18 passed

vitals

time

30m 03s

answered

100%

failed

success

100%

systems

Hard coding test v2

18/18 passed

30m 03s

fix-one-1✓ pass3m 18s
prompt
Download airbench.ai/f/9983a4d9a7034986cc198487a2c0a11f.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Routine: classic mutable default argument (lots=[]) shared across bins; fixed with None default, all tests pass.
trace-1✓ pass22s
prompt
What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. function v1f(x = y, y = 47) { return x + y; } const v1safe = (fn) => { try { return fn(); } catch (e) { return "TDZ"; } }; const v1 = [v1f(1), v1safe(() => v1f()), v1f(1, 2), v1safe(() => v1f(undefined, 1))].join(","); let v2 = ""; switch (1) { case 3: v2 += "a"; case 4: v2 += "b"; case 1: v2 += "c"; break; default: v2 += "d"; case 5: v2 += "e"; } function v3f() { let log = ""; for (let i = 0; i < 2; i++) { try { if (i === 0) continue; return 4; } finally { log += i; if (i === 1) return log + 42; } } } function v3g() { try { throw new Error("x"); } catch (e) { return 4; } finally { return 42; } } const v3 = v3f() + "," + v3g(); const v4m = [4,,9].map(x => x * 2); const v4 = v4m.join("-") + ":" + v4m.length; function* v5gen() { try { yield 9; } catch (e) { yield e + 12; } yield 28; } const v5it = v5gen(); v5it.next(); const v5x = v5it.throw(128).value; const v5y = v5it.next(); const v5 = v5x + "," + v5y.value + "," + v5it.next().done; console.log(v1, v2, v3, v4, v5);
answer
answer hidden on shared reports
agent's debrief
Traced by hand (running node was blocked by my sandbox's permission check). Key points: default-param TDZ, switch fallthrough, finally overriding return, holes preserved by map, generator throw. Fairly confident.
spec-impl-1✓ pass11s
prompt
Specification: tiered pricing. Write a function price(units) that returns the cost, in whole cents, of buying `units` units (a non-negative integer) under marginal pricing: - the first 9 units cost 90 cents each; - units 10 through 39 (inclusive) cost 50 cents each; - every unit after the 39th costs 35 cents each. Each bracket's rate applies only to the units that fall inside that bracket (it is not retroactive to earlier units). Add the three bracket subtotals, then round the total to the nearest multiple of 10 cents. A total that ends in exactly 5 sits halfway between two multiples of 10: round it up (for example 1235 becomes 1240). Any other total rounds to the closer multiple of 10. Zero units cost 0. For each input line, which holds `units`, output the rounded total as a plain integer number of cents (no currency symbol, no decimal point). The 15 inputs, one per line: 0 9 10 40 39 4 27 10 11 52 22 67 68 2 46 Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).
answer
answer hidden on shared reports
grader
15/15 lines match
agent's debrief
Easy: straightforward marginal brackets plus round-half-up to 10 cents.
issue-crash-1✓ pass24s
prompt
Download airbench.ai/f/9fdf4ba28a91a8eb61b2ad4362c7620a.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > Some products are in categories the report has never heard of, and the README says every category other than books is taxed. > ``` > Traceback (most recent call last): > File "/home/user/purchases/main.py", line 21, in <module> > main() > File "/home/user/purchases/main.py", line 13, in main > orders = collect_orders(rows) > File "/home/user/purchases/checkout/baskets.py", line 34, in collect_orders > total_order(order) > File "/home/user/purchases/checkout/baskets.py", line 41, in total_order > order.tax = order_tax(order.lines, order.region) > File "/home/user/purchases/checkout/tax.py", line 21, in order_tax > if not is_exempt(line.category): > File "/home/user/purchases/checkout/tax.py", line 12, in is_exempt > kind = CATEGORY_KIND[cat] > KeyError: 'garden' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Easy: unknown categories raised KeyError; changed lookup to default to taxed per README. Sample checksum still matches. Did not audit the rest of the code for unrelated README deviations.
issue-feature-1✓ pass32s
prompt
Download airbench.ai/f/b7341718d3a19bfd1026151c6628a999.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Support the new `discount_pct` column** > > Our export now has an optional last column, `discount_pct`: a whole percentage, where blank means 0. Apply it to the line's base first: `discounted = base - base * discount_pct // 100` (rounded down). Then the volume discount of README rule 5 applies to that discounted amount, exactly as it applies to `base` today (the tier still depends on the quantity). Everything else stays as the README says. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category,discount_pct > D8801,2026-03-14T10:00:00Z,c733,EU,EL-552,12,1.01,EUR,general,15 > D8802,2026-03-14T10:00:00Z,c733,EU,FD-119,3,4.00,EUR,food, > ``` > > `python main.py example.csv` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=2 gross=27.13 rate=25% last=D8802 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=2 gross=29.25 rate=25% last=D8802 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Routine: added optional discount_pct to the row, applied before the volume tier; the example in the request reproduces exactly (27.13). Sample checksum changes as the prompt warned, since sample has the column.
spec-violation-1✓ pass23s
prompt
Download airbench.ai/f/103ef271e7b06937b96e4747437b1065.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Easy: volume tiers used > instead of >= (README says 8 or more / 60 or more). Fixing that made the sample checksum match.
far-cause-1✓ pass28s
prompt
Download airbench.ai/f/38fd5977791efc3b5db0431bf64be817.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `LATAM 2025-12 JPY` the ledger has a gross of `1033547`, but the report prints `gross=827715`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Moderate: the timestamp parser added the UTC offset instead of subtracting it, moving offset rows across month boundaries. After the fix LATAM 2025-12 JPY shows the ledger's 1033547 and the sample checksum still matches.
interacting-1✓ pass29s
prompt
Download airbench.ai/f/10d561b57543cd0001664a786a73b293.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `EU 2026-01 EUR` | `303892.36` | `304315.83` | > | `LATAM 2026-01 JPY` | `3495692` | `34938.59` | > | `US 2025-11 USD` | `71089.83` | `69982.60` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Moderate: found three bugs by reading against the README (JPY treated as 2 decimals with a //100 hack, revisions compared by raw ts string instead of UTC instant, tier thresholds > instead of >=). All three ledger groups now match exactly.
perf-1✓ pass1m 35s
prompt
Download airbench.ai/f/9defb52c67bc277a48d2ab86f657551c.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 811799648 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.
answer
answer hidden on shared reports
agent's debrief
Routine: replaced four quadratic spots (selection-sort percentile, list-membership pair count, all-pairs concurrency, full max over totals per event) with sort, set, sweep line and an incremental leader check (valid since values are non-negative). Sample checksum matches and a 20k-event log matches the original; 1M run takes ~5s.
rev-cpu-target1✓ pass9m 14s
prompt
This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/0835eb9ecc9ff39e8162529665100a3c.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>
answer
answer hidden on shared reports
agent's debrief
Built an emulator from the probes (56 opcodes incl. carry/no-flags prefix bytes, flag-word layout, interrupt vectors at FD00, bitfield extract, and a rotr3^x+0x2C4F mixer); it reproduces all 8 probe traces step-for-step. Target ran to halt with readable output, so fairly confident.
rev-cpu-target2✓ pass2m 19s
prompt
This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/0835eb9ecc9ff39e8162529665100a3c.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>
answer
answer hidden on shared reports
agent's debrief
Same emulator (all probe traces reproduce exactly). Target 2 prints big-number arithmetic and hex; output looks plausible (factorial-like digits ending in zeros). Confident if no untested edge semantics were hit.
rev-cpu-target3✓ pass6s
prompt
This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/0835eb9ecc9ff39e8162529665100a3c.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>
answer
answer hidden on shared reports
agent's debrief
Same probe-validated emulator. Target 3 prints a grid of hex words; no way to cross-check beyond the probes, but nothing anomalous happened.
rev-cpu-target4✓ pass8s
prompt
This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/0835eb9ecc9ff39e8162529665100a3c.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>
answer
answer hidden on shared reports
agent's debrief
Target 4 performs one divide-by-zero; I modelled it from probe 6 (registers unchanged, V set, other flags kept). Moderately confident; that is the main risk.
rev-cpu-target5✓ pass9s
prompt
This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/0835eb9ecc9ff39e8162529665100a3c.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>
answer
answer hidden on shared reports
agent's debrief
Target 5 appears to exercise interrupts/flag words; the vector table at FD00 and the flag-word bit layout (C=20,Z=40,N=80,V=04) were validated on the probes. Fairly confident.
rev-cpu-target6✓ passbatched
prompt
This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/0835eb9ecc9ff39e8162529665100a3c.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>
answer
answer hidden on shared reports
agent's debrief
Same probe-validated emulator; target 6 ran to halt with hex-word output. Fairly confident.
opt-route-bronze✓ pass10m 23s
prompt
This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/ff9c14f19fe71998239ecedeb24a1de6.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
answer
answer hidden on shared reports
grader
valid, band 3/3
agent's debrief
ALNS in C++ (random/Shaw/worst/route removal, greedy+regret insertion of whole groups, simulated annealing, then per-route start-time optimisation). Best of 6 seeds x 6 min: cost 166116 with 41 routes, checked with my own validator using the reference toll code. Capacity bound is 38 vehicles, so there is likely headroom; unsure how it compares to the reference.
opt-route-silver✓ passbatched
prompt
This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/ff9c14f19fe71998239ecedeb24a1de6.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
answer
answer hidden on shared reports
grader
valid, band 3/3
agent's debrief
ALNS in C++ (random/Shaw/worst/route removal, greedy+regret insertion of whole groups, simulated annealing, then per-route start-time optimisation). Best of 6 seeds x 6 min: cost 166116 with 41 routes, checked with my own validator using the reference toll code. Capacity bound is 38 vehicles, so there is likely headroom; unsure how it compares to the reference.
opt-route-gold✓ passbatched
prompt
This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/ff9c14f19fe71998239ecedeb24a1de6.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
answer
answer hidden on shared reports
grader
valid, band 3/3
agent's debrief
ALNS in C++ (random/Shaw/worst/route removal, greedy+regret insertion of whole groups, simulated annealing, then per-route start-time optimisation). Best of 6 seeds x 6 min: cost 166116 with 41 routes, checked with my own validator using the reference toll code. Capacity bound is 38 vehicles, so there is likely headroom; unsure how it compares to the reference.

what the agent said about this checkup

Overall: all 18 challenges answered, none marked unable. Environment note: my sandbox's auto-mode safety check blocked running downloaded code (and even node on a pasted snippet) until the user approved it; so the JS trace (challenge 2) was done purely by hand. Easy/routine: fix-one (mutable default arg), tiered pricing, issue-crash (unknown category KeyError -> default taxed), spec-violation (> vs >= tiers), far-cause (UTC offset added instead of subtracted; ledger value then matched exactly), feature (discount_pct; the worked example reproduced exactly), interacting (three bugs: JPY decimals hack, revisions compared by raw ts string, tier thresholds; all three ledger groups then matched exactly), perf (four quadratic spots; verified identical checksum vs the original on a 20k-event log and the sample). Hard: the undocumented CPU. Most opcodes fell out of the traces quickly, but several things took real work: operand order of the store-postincrement ops, the flag-word bit layout (C=0x20,Z=0x40,N=0x80,V=0x04), that 0x71 is a one-shot 'use carry' prefix and 0xEB a one-shot 'preserve flags' prefix, that INC/DEC keep carry, logic ops keep C and V while shifts clear V, shift counts masked to 4 bits, F3 being a bitfield extract, E9 an indirect jump, interrupt vectors at FD00, divide-by-zero behaviour, and opcode 0x80 = (rotr(x,3) ^ y) + 0x2C4F which I only found by noticing the low 13 bits tracked x>>3. Final emulator reproduces all 8 probe traces step-for-step (regs, flags, memory writes, output). Residual risk: target behaviour on any opcode/prefix combination not exercised by the probes (target4 does one divide-by-zero, modelled from probe6). Routing: wrote a C++ ALNS; best valid plan 166116 with 41 routes after ~6 min x 6 seeds. Validated with my own checker based on the reference toll code. I don't know the reference cost; capacity lower bound is 38 vehicles so silver/gold may well fail. With more time I'd add a dedicated route-minimisation phase and intra-route local search. Nothing seemed broken or unfair; the tasks were clearly specified.

in the agent’s own words · not graded

discussion

No messages yet.