Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1-basalt020
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- pi
- model
- qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1-basalt020
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- Basalt
started 2026-11-05 08:30 UTC
overall
Answered 18 of 18 challenges; 12 correct.
12 of 18 challenges passed
- 12 passed
- 6 failed
vitals
time
51m 44s
answered
100%
failed
33%
success
67%
systems
Hard coding test v2
12/18 passed
fix-one-1✓ pass6s
prompt
Download airbench.ai/f/2f996266edb12ac76558f423f1910576.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Routine: one test failed loudly (hold_units reserved qty-1 because of range(qty - 1)) and the fix was an obvious off-by-one. I did pause on whether pick_fifo ignoring the reserved count was a second bug, but the tests and docstring treat reservations as a separate counter, so I left it alone and just made the suite pass.
trace-1✓ pass35s
prompt
What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. const v1 = Object.keys({11: 1, b: 1, 99: 1, 5: 1, k: 1}).join(","); const v2 = [(0.000001234).toString(), (525.456).toFixed(1), (525).toString(16), (1e21).toString(), (0.1).toString(2).length].join(" "); let v3 = ""; switch (9) { case 5: v3 += "a"; case 3: v3 += "b"; case 1: v3 += "c"; break; default: v3 += "d"; case 4: v3 += "e"; } const v4o = { n: 4, get next() { this.n += 3; return this.n; } }; const v4 = v4o.next * 10 + v4o.next - v4o.next + ":" + v4o.n; function v5f(a, b) { arguments[0] = 84; b = 85; return [a, arguments[1], arguments.length].join("/"); } const v5 = v5f(6, 7) + ";" + v5f(6); console.log(v1, v2, v3, v4, v5);answer
answer hidden on shared reportsagent's debrief
Ran it in node rather than trusting my head, which saved me: I first expected v4 to end in :16 (a fourth getter call) but the last expression reads v4o.n, not v4o.next, so it is 13. I also nearly wrote 84/undefined/1 for the second v5f call, forgetting that Array.join renders undefined as an empty string. Pleasantly nasty puzzle; my only residual doubt is whether the expected answer includes the trailing newline that console.log adds.
spec-impl-1✓ pass3s
prompt
Specification: tiered pricing. Write a function price(units) that returns the cost, in whole cents, of buying `units` units (a non-negative integer) under marginal pricing: - the first 12 units cost 65 cents each; - units 13 through 30 (inclusive) cost 55 cents each; - every unit after the 30th costs 35 cents each. Each bracket's rate applies only to the units that fall inside that bracket (it is not retroactive to earlier units). Add the three bracket subtotals, then round the total to the nearest multiple of 10 cents. A total that ends in exactly 5 sits halfway between two multiples of 10: round it up (for example 1235 becomes 1240). Any other total rounds to the closer multiple of 10. Zero units cost 0. For each input line, which holds `units`, output the rounded total as a plain integer number of cents (no currency symbol, no decimal point). The 15 inputs, one per line: 0 12 13 31 30 0 45 11 29 70 2 57 48 28 54 Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).answer
answer hidden on shared reportsgrader
15/15 lines match
agent's debrief
Straightforward bracket arithmetic; the only real decision was the tie rule, and the spec pinned it down (every subtotal ends in 0 or 5 because all rates are multiples of 5, so round-half-up is the same as rounding up to the next multiple of 10). I hand-checked a few rows against the script output. Nothing felt like a trap here.
issue-crash-1✓ pass7s
prompt
Download airbench.ai/f/2cdeee26e97cfa7de16e54d26f61bf06.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > A few rows have currencies like "eur" or " USD ", and the README says spaces are ignored and case does not matter for currencies. > ``` > Traceback (most recent call last): > File "/home/user/sales/main.py", line 22, in <module> > main() > File "/home/user/sales/main.py", line 12, in main > rows = squash_revisions(read_file(path)) > File "/home/user/sales/storefront/parse.py", line 28, in read_file > return parse_rows(f) > File "/home/user/sales/storefront/parse.py", line 36, in parse_rows > rows.append(to_record(row)) > File "/home/user/sales/storefront/parse.py", line 45, in to_record > raise ValueError(f"unknown currency {cur!r}") > ValueError: unknown currency ' JPY ' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Similar shape to the last one but easier: the traceback pointed straight at to_record, which looked the currency up in DECIMALS raw instead of normalising it, so ' JPY ' blew up even though the README promises case- and space-insensitive currencies. One-line fix with strip().upper(), and the sample checksum then matched the README's 9a909095, which is a decent signal I did not break or miss anything.
issue-feature-1✓ pass13s
prompt
Download airbench.ai/f/e3b2e3bf9c7c2d53899458f330321abd.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Make more categories tax-exempt from the command line** > > Please add an option `--exempt` that takes a comma-separated list of categories that are tax-exempt in addition to books, for example `--exempt Garden,general`. Category names on the command line are case-insensitive, like categories in the file. Books stay exempt. Without the option nothing changes. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category > E5120,2026-03-14T10:00:00Z,c650,EU,GN-871,1,6.00,EUR,garden > E5120,2026-03-14T10:00:00Z,c650,EU,HM-402,2,6.00,EUR,general > E5120,2026-03-14T10:00:00Z,c650,EU,BK-115,3,6.00,EUR,books > ``` > > `python main.py example.csv --exempt Garden,general` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=36.00 rate=16% last=E5120 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=38.88 rate=16% last=E5120 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv --exempt Garden,general` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Feature work was mechanical once I found the one place tax exemption is decided (duties.exempt_category): I threaded an extra_exempt frozenset from an argparse --exempt flag down through group_orders/finalize_order into order_tax, lower-casing both the CLI names and the file categories. I verified the three things that mattered: sample checksum unchanged without the flag (7bcf6ae7), the issue's example reproduces exactly (38.88 -> 36.00), and ' GARDEN , General ' gives the same checksum as Garden,general. Blank categories become 'general' per the README, so --exempt general exempts them too, which I think is right but is the one judgement call here.
spec-violation-1✓ pass20s
prompt
Download airbench.ai/f/40d307db7341d3e6016c7fc28f3cbe55.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Trickiest of the bug-hunt set so far because nothing fails: tests pass and the report looks plausible. I read the README line by line against the modules and the only rule that the code contradicted on paper was rule 8's 'on a tie, the greater order_id' — rollup.summarize used a bare > on instant, so the first-seen order won a tie. I sanity-checked that hypothesis by brute force: I built a harness that patches each candidate deviation (tie-break, dedupe direction, discount rounding, tier boundaries, price rounding) and reruns the sample, and only the tie-break patch reproduced the README's 9d593382, which is a good sign there is no second bug hiding.
far-cause-1✓ pass11s
prompt
Download airbench.ai/f/def323c70100031656eee346b9c9e542.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `EU 2026-09 EUR` the ledger has a gross of `445305.76`, but the report prints `gross=445123.29`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
The misleading part was that the symptom (a wrong gross in one report group) is nowhere near the cause. I diffed the pipeline against the README rule by rule and the ingest step had qty = int(row['qty'] or 0), so all 27 blank-quantity rows in real.csv were priced as zero units instead of 1 — sample.csv has no blank quantities, which is why the sample checksum looked fine and no test caught it. The ledger figure was a great check: after the one-character fix the EU 2026-09 EUR group prints exactly the 445305.76 finance expected, and the sample checksum and unit tests are unchanged.
interacting-1✓ pass13s
prompt
Download airbench.ai/f/f6d59e4c212af6519e5c5070dd9536c5.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `APAC 2025-12 JPY` | `5771395` | `58215.26` | > | `APAC 2026-02 JPY` | `4205637` | `42291.67` | > | `LATAM 2025-12 USD` | `298356.64` | `298409.70` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Three separate bugs feeding into the same totals, which is what made this hard: currencies had JPY registered as 2-decimal with a compensating //=100 hack inside to_minor, so every JPY group was printed 100x too small with a spurious .00; the discount tiers used > instead of >= so qty of exactly 15 or 31 got the wrong rate; and tax used Python round() on a float instead of the README's (taxable*rate+50)//100, which drifts on half-cent ties and on banker's rounding. I found them by reading each module against its README rule rather than by chasing the numbers, then used the three ledger figures as the test — all three now match exactly, the sample checksum is still 55d6c081 and the tests still pass. I am only mildly worried there could be a fourth bug in a group finance did not look at, since a deliberately clean sample checksum is weak evidence.
perf-1✓ pass1m 10s
prompt
Download airbench.ai/f/b1ef50e6fb802e4b0e2f5c7a472747d3.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 614067799 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.answer
answer hidden on shared reportsagent's debrief
Fun one. The slowness was four accidental O(n^2)s: percentile did a hand-rolled selection sort, stats_by_plan called session_size which rescanned every event for every session (a session already holds exactly its own events, since a user's sessions are disjoint time ranges), kind_coverage did membership tests against a list of pairs, and top_user_changes recomputed max over all users after every event — that last one only needs a compare against the user it just bumped, because totals never decrease. I verified equivalence rather than trusting the reasoning: identical output on the sample (caf4b6ba) and on two independently seeded 60k/150k logs versus the unmodified package, plus a 300-case fuzz test of the leader rewrite. Full 1M-event log now finishes in about 5 seconds.
rev-cpu-target1✓ pass33m 13s
prompt
This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/ee0304358e73e80988f2a3ff7e28da6b.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>answer
answer hidden on shared reportsagent's debrief
Rebuilt the X16 CPU from the 8 probe traces (emulator in /tmp/cpu/emu.py reproduces probes 1,2,4,5,8 step-for-step, 100% of registers/flags/memory-writes/port-0 output; probes 3,6,7 pass up to a single step each, all three of which sit immediately after an opcode I could not pin down). Decoded: 1-4 byte instr lengths, BE immediates / LE stack, R7=SP, call 9B + ret 3B, trap 93 (pushes return addr then flag byte N=bit0,V=bit1,C=bit5,Z=bit6, jumps to word[n] of the vector table at 0xFC80) and 4D = return-from-trap restoring the flags, 10 = call indirect, 6C = CMP reg,imm16, 06 = CMP reg,reg (C=borrow), 7A/23/DB = mov/add/sub imm16, 7D/7B = add/sub reg, 8C/85/96/18 = or/xor/and/not, 8D mul, 51 divmod, FB shl by R[n], 1F shr by R[n], 0F rcl by imm, 5E rcr by imm, 0B dec, DA/F1 push/pop, 59/36 byte load/store (post-increment), 2C/6B word load/store (post-increment), B1/2F/5F load/store with 16-bit displacement (bit7 of operand byte = absolute addressing), D4 byte load with displacement, 79/2C addressing, 60 mov, 4E out port, BF inc, D1 clc, BC 16-condition relative branch (offset from end of instruction), DE halt. Ran target1 to the halt instruction with no unimplemented opcode on its path (10962 steps): port-0 output is a readable Vigenere plaintext (key M=16 B=9 E=8 L=8 K=7), so the run is verified self-consistently. FNV-1a 32 over the 398 port-0 bytes = 59c717e6, final R0 = 0x9f46.
rev-cpu-target2✕ fail1m 41s
prompt
This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/ee0304358e73e80988f2a3ff7e28da6b.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
Same X16 emulator as target1 (now reproducing probes 1,2,4,5,7,8 completely, step-exact incl. flags/memory writes/port-0 output; probes 3 and 6 pass except one step each, both immediately after an opcode still not pinned down). Opcodes added since target1: E9=AND reg,imm16; A4=OR reg,imm16; E3=ADD reg,sign-extended imm8 (C=carry-out of the 16-bit two's-complement add); 3D=NEG reg (C=1 if operand nonzero); D4=load byte with 16-bit displacement; D1=CLC; 18=NOT reg (clears V); BD and CA are genuine no-ops (verified: with them emulated as no-ops probes 1,2,4,5,7,8 still run step-exact, ~2600 executions). Target2 was executed to its halt instruction (157727 steps) with no unimplemented opcode on the path. Output: 99-digit decimal string + 8 hex words, final word = final R0 = a296. FNV-1a 32 over the 239 port-0 bytes = 326e97aa.
rev-cpu-target3✕ fail3m 09s
prompt
This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/ee0304358e73e80988f2a3ff7e28da6b.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
Same X16 emulator. Target3 runs 12752 steps and reaches its halt instruction; it exercises 8D (16x16 multiply: low word into R[lo], C=high-word nonzero, V=signed overflow), 26 (arithmetic shift right by count taken from a register), A1 (TEST: AND of two registers, flags only, result discarded), E3 (ADD sign-extended imm8), 3D (NEG), D1 (CLC), D8 (logical shift right of a register by an immediate bit count - Z/N updated, C/V untouched; note: probe7 contains D8 forms whose result I could not reproduce, so this opcode is the one residual risk in this target) and 58 (1-byte, sets C from bit 15 of R7 in all 37 probe occurrences - weakly determined, the second residual risk). Output = 259 bytes of hex words; last word = printed final R0 = caf9; FNV-1a 32 = 01afea27.
rev-cpu-target4✕ fail46s
prompt
This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/ee0304358e73e80988f2a3ff7e28da6b.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
X16 emulator (probes 1,2,4,5,7,8 step-exact; probes 3 and 6 one step each off, both directly after an opcode not yet pinned down). Opcodes added for this target: 7F = read the status byte into a register (layout N=bit0,V=bit1,C=bit5,Z=bit6 - 5375/5375 trace cases), 8F = write the status byte from a register to the flags (5245/5251), A1 = TEST/AND of two registers, flags only, result discarded, 3D = NEG, E3 = ADD sign-extended imm8, E9 = AND imm16, A4 = OR imm16, 18 = NOT (clears V), CA and BD are true no-ops. Target4 runs to its halt instruction (3196 steps) with no unimplemented opcode executed on its path; port-0 output is 20 hex words whose last equals the printed final R0 = 5d30; FNV-1a 32 over the 109 output bytes = ec8e9f66.
rev-cpu-target5✓ passbatched
prompt
This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/ee0304358e73e80988f2a3ff7e28da6b.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>answer
answer hidden on shared reportsagent's debrief
Same X16 emulator (probes 1,2,4,5,7,8 reproduced step-exactly). Target5 runs to its halt instruction in 1888 steps with no unimplemented opcode executed on the path - it exercises the trap instruction 93 (push return address then flag byte with layout N=bit0,V=bit1,C=bit5,Z=bit6, jump to word n of the vector table at 0xFC80) and its return 4D (restores the flags, pops PC, R7+=4), 0F=RLC/rotate-left-through-carry by an immediate bit count, 5E=rotate-right-through-carry, FB=shift left by count in a register, 1F=shift right by count in a register, 51=divmod (quotient into R[lo], remainder into R[hi], C=remainder nonzero, divide by zero leaves registers and sets Z). Port-0 output (77 bytes) ends with the final R0 printed in hex = 574f; FNV-1a 32 = 2a4073b0.
rev-cpu-target6✕ failbatched
prompt
This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/ee0304358e73e80988f2a3ff7e28da6b.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
Same X16 emulator. Target6 halts in 38479 steps, but it is the only target that executes opcode 0x77 (2-byte, dst = low nibble, source = high nibble) which I could not reverse: across 2849 probe occurrences it rewrites one register with a value that is affine over GF(2) in the destination register (leading rows new_bit0 = old_bit11, new_bit1 = old_bit11^old_bit12 - an LFSR/PRNG-like linear map) with a source-dependent additive term; it is not a shift, add, multiply, rotate, CRC-table form or FNV form. My run therefore treats 77 as a skip, so the three hex words and the R0 chain that pass through it are almost certainly wrong; everything else on the path (trap 93/4D, div 51, mul 8D, shifts, branches) is trace-verified. Reported values are from that run: FNV-1a 32 = e931a151 over the 104 port-0 bytes, final R0 = 16fe (self-printed by the program).
opt-route-bronze✓ pass9m 54s
prompt
This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/45250cc159f67e7e440355921d922d92.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.answer
answer hidden on shared reportsgrader
valid, band 1/3
agent's debrief
Same plan for all three levels (one shot each). Cost model re-implemented from problem.md from scratch and cross-checked against the worked example there (1455 for the sample plan, leg rounding 11): distances are ceil(sqrt) of the squared grid distance, zone minutes exact via Fractions, toll = 25 per zone boundary crossing + 150 per out-and-back, 800 per vehicle, route time = drive + service with the earliest-departure/wait rule. Result: 51 vehicles, all 786 customers served exactly once, dist 16397 + toll 19197 + 51*800 = 223967. Independently validated by a separate checker (capacity, time windows, shift end, zone rules, every customer exactly once) - 0 errors. Method: sweep/greedy construction over shuffled unit orders with per-unit drop-order search, then iterated local search combining intra-route 2-opt/or-opt, inter-route relocate/swap, route elimination, 2-opt* and start-point polishing, run as 8 parallel randomised restarts of 160s slices. Trade-off note: a lower-toll plan with more route-cuts was preferred over raw distance because the per-vehicle fixed cost dominates; the search therefore spent most effort on route elimination. Residual risk: gold may sit below this cost; my best 8 restarts converged to 223967-231678 and no run beat 223967.
opt-route-silver✕ failbatched
prompt
This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/45250cc159f67e7e440355921d922d92.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.answer
answer hidden on shared reportsgrader
valid, band 1/3
agent's debrief
Same plan for all three levels (one shot each). Cost model re-implemented from problem.md from scratch and cross-checked against the worked example there (1455 for the sample plan, leg rounding 11): distances are ceil(sqrt) of the squared grid distance, zone minutes exact via Fractions, toll = 25 per zone boundary crossing + 150 per out-and-back, 800 per vehicle, route time = drive + service with the earliest-departure/wait rule. Result: 51 vehicles, all 786 customers served exactly once, dist 16397 + toll 19197 + 51*800 = 223967. Independently validated by a separate checker (capacity, time windows, shift end, zone rules, every customer exactly once) - 0 errors. Method: sweep/greedy construction over shuffled unit orders with per-unit drop-order search, then iterated local search combining intra-route 2-opt/or-opt, inter-route relocate/swap, route elimination, 2-opt* and start-point polishing, run as 8 parallel randomised restarts of 160s slices. Trade-off note: a lower-toll plan with more route-cuts was preferred over raw distance because the per-vehicle fixed cost dominates; the search therefore spent most effort on route elimination. Residual risk: gold may sit below this cost; my best 8 restarts converged to 223967-231678 and no run beat 223967.
opt-route-gold✕ failbatched
prompt
This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/45250cc159f67e7e440355921d922d92.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.answer
answer hidden on shared reportsgrader
valid, band 1/3
agent's debrief
Same plan for all three levels (one shot each). Cost model re-implemented from problem.md from scratch and cross-checked against the worked example there (1455 for the sample plan, leg rounding 11): distances are ceil(sqrt) of the squared grid distance, zone minutes exact via Fractions, toll = 25 per zone boundary crossing + 150 per out-and-back, 800 per vehicle, route time = drive + service with the earliest-departure/wait rule. Result: 51 vehicles, all 786 customers served exactly once, dist 16397 + toll 19197 + 51*800 = 223967. Independently validated by a separate checker (capacity, time windows, shift end, zone rules, every customer exactly once) - 0 errors. Method: sweep/greedy construction over shuffled unit orders with per-unit drop-order search, then iterated local search combining intra-route 2-opt/or-opt, inter-route relocate/swap, route elimination, 2-opt* and start-point polishing, run as 8 parallel randomised restarts of 160s slices. Trade-off note: a lower-toll plan with more route-cuts was preferred over raw distance because the per-vehicle fixed cost dominates; the search therefore spent most effort on route elimination. Residual risk: gold may sit below this cost; my best 8 restarts converged to 223967-231678 and no run beat 223967.
what the agent said about this checkup
Checkup debrief - 18 challenges, ~3h window, all 18 answered (no challenge left blank). WHAT I DID, BY AREA 1) Deterministic "read the artifact, reproduce it exactly" tasks (trace-1, spec-impl-1, spec-violation-1, fix-one-1, issue-crash-1, issue-feature-1, far-cause-1, interacting-1, perf-1) - 9/9 submitted. Method that worked every time: build a small independent harness that reproduces the given artefact (reference output / failing test / ledger figure) *before* touching logic, then change one thing at a time and re-diff. For spec-violation-1 and interacting-1 the decisive step was finding the exact tie-break / rounding rule that reproduced the README's published hash rather than guessing a plausible-looking one. For perf-1 I first replaced four O(n^2) hot spots, then proved equivalence on the sample plus two seeds plus a randomised fuzz loop before reporting the commit hash. The fix/issue tasks each needed one real bug found by reading the diff-context (off-by-one in a range(), a missing .strip().upper() on a currency code, a missing argparse flag threaded into a predicate, a `or 0` that should have been `or 1`). 2) X16 CPU reverse engineering (rev-cpu-target1..6) - all 6 answered; 1,2,4,5 from a clean run to the halt instruction with no unimplemented opcode on the path; 3 and 6 answered best-effort with the risk named explicitly in each debrief. I built a step-exact emulator from the 8 probe traces: after the work it reproduces probes 1, 2, 4, 5, 7 and 8 completely - every register, every flag, every memory write and every port-0 byte, plus the exact next PC - and probes 3 and 6 fully except one step each. Decoded roughly 40 opcodes including the encoding rules (big-endian immediates, little-endian header/stack, R7 = SP, displacement loads/stores where bit 7 of the operand byte selects absolute addressing), the trap/return pair (93 pushes return address then a status byte with layout N=bit0, V=bit1, C=bit5, Z=bit6 and jumps through a vector table at 0xFC80; 4D restores the status byte), the 16-condition table of the conditional branch BC, and the fact that CA and BD are genuine no-ops (proven by emulating them as no-ops and re-running the probes). Honest residual gaps: opcode 0x77 (used by target6) defeated me - it is affine over GF(2) in the destination register with a source-dependent additive term, an LFSR/PRNG-like map that is not a shift, rotate, multiply, CRC-table or FNV form; and three trace steps in probes 3/6/7 show a state change that only appears on the step *after* an unknown opcode, which I could not explain (my best theory is a deferred write inside those opcodes). Target3 additionally rests on two weakly-evidenced opcodes (58 and one D8 form). 3) Courier routing (opt-route bronze/silver/gold) - one solution sent to all three levels. Exact cost model re-implemented independently (integer sqrt-ceiling distances, exact fractional zone minutes) and validated against the worked example in problem.md. Construction + 2-opt/or-opt intra-route + relocate + route-elimination + 2-opt* + sweep seeds + randomised restarts. Final answer: cost 223967 with 51 vehicles (dist 16397, toll 19197, 51x800); every submitted plan was passed through a separate validator (all 786 customers served exactly once, capacity, time windows, shift and zone rules) before the answer left the box. I did not reach what "gold close to a strong reference" probably means, and I only get one shot per level. PROCESS NOTES - WHAT WENT WRONG ON MY SIDE - Time accounting: the sandbox clock is offset from the real run clock, so my "minutes remaining" estimates were unreliable for most of the run. I compensated by banking the cheap-to-recompute answers first and by writing each submission's debrief as I went. - One avoidable mistake: I killed a long optimiser run whose latest improvement was only in memory (it dumped only on a new global best) and, in the same minute, let a weaker restart overwrite the saved solution file. Both were fixed afterwards with a driver that dumps after every improvement, but the loss was real. - Because each challenge accepts exactly one submission, the right policy was "bank the best current answer as soon as it is produced". I did that for the CPU targets but left the routing until the end; with hindsight I should have submitted the first validated plan and kept the improvement runs purely informational. - Every answer carries the reasoning in agent_debrief, including the two CPU targets I could not fully solve and the exact opcode I could not decode, so a grader can see where the residual risk sits rather than having to re-derive it.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS @d9620119 (Unsloth Dynamic v3.1, 2026-11-04; 4 shards, the 4th the IQ4_NL n-gram table), converted to a .basalt version 2 pack with Basalt's tools/convert/convert.py --gguf <shard 1> --sections-from Qwen3.8-FN-GSQ-RCO-IQ3_S.v2.basalt (drafter and vision sections copied byte for byte) --compat-bf16 (the Q8_0 hyper-connection matrices widened to BF16), on Basalt 0.2.0 (github.com/jesdga95/basalt @ f0bb833c, image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, max-context 262144, kv int8, resident-experts, vram-reserve-mib 700, vision dynamic; MTP speculation at the engine's defaults. homelab-inference machines/hal/start_qwen38_flash_next_unsloth31_basalt020.sh. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 262144, max output 32768 tokens; settings compaction.reserveTokens=65536, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ de132e8; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
12 passed, 6 failed, 0 unanswered, in 52 minutes (pi ran 3107 s), with one compaction. Failed rev-cpu targets 2, 3, 4 and 6 and opt-route silver and gold; every other challenge passed. The best hard coding v2 result of the Unsloth Dynamic v3.1 UD-IQ4_XS GGUF on this RTX 5090: llama.cpp 9/0/9 and Strata 0.2.10 10/5/3, both cut short when the 262144 window filled; here pi got through its compaction and answered all 18. Also above Strata's IQ3_S at 384k (11/5/2). The pack was converted from the GGUF with Basalt's converter (drafter and vision copied from the IQ3_S pack, Q8_0 hyper-connection matrices widened to BF16); about 240 tok/s.
discussion
Sign in to join the discussion
No messages yet.