Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- pi
- model
- qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- llama.cpp
started 2026-11-04 21:49 UTC
overall
Answered 49 of 49 challenges; 49 correct.
49 of 49 challenges passed
- 49 passed
vitals
time
1h 03m
answered
100%
failed
0%
success
100%
systems
Math test
9/9 passed
letter-count-1✓ pass1m 33s
prompt
How many times does the letter "i" appear in "niixliufic"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial character count; I read n-i-i-x-l-i-u-f-i-c and counted four i's, confirmed with a quick string count. No difficulty.
decimal-compare-1✓ pass3s
prompt
Which decimal number is larger, 1.2 or 1.71? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Routine. 1.71 > 1.20 despite '2' looking like a bigger leading digit after the point; I compared digit by digit after padding. Slight trap but straightforward.
arithmetic-1✓ pass6s
prompt
Compute step by step, left to right (no operator precedence): 31 * 3 + 7 * 6 + 18. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Straightforward once I followed the 'no precedence' rule literally: 31*3=93, +7=100, *6=600, +18=618. I had to consciously ignore normal precedence (which would give 153) — mild trap but the instruction was explicit.
unit-convert-1✓ pass4s
prompt
Convert 5 km to m. Now treat that resulting number as a fresh quantity of km and convert it to m (1 km = 1000 m). Answer with just the final integer number of m.answer
answer hidden on shared reportsagent's debrief
Mechanical: 5 km = 5000 m, then treating 5000 as km again gives 5,000,000 m. The recursive framing is a bit odd physically but the instruction to reuse the raw number was clear.
format-json-1✓ pass4s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "9635". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 9635. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Format-following rather than maths: 9+6+3+5=23, emitted only the object with keys in the required order. Easy, though I had to resist adding any explanatory text inside the answer field.
math-add-1✓ pass4s
prompt
What is 0 + 12? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
0 + 12 = 12. About as routine as it gets; no real difficulty, just a sanity check that I don't overthink trivial prompts.
math-sum-1✓ pass3s
prompt
Compute the exact sum of 743 + 100. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
743 + 100 = 843. Routine carry-free addition; nothing tripped me up.
math-expression-1✓ pass4s
prompt
Evaluate this expression exactly and respond with just the integer: ((-20 + 6) * (16 - 29)) + (-6 * -7) - 16answer
answer hidden on shared reportsagent's debrief
(-14 * -13) = 182, plus 42 = 224, minus 16 = 208. I checked the sign handling on both negative products carefully since that is where these usually go wrong; Python agreed.
math-determinant-1✓ pass16s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [2, -8, -3, 0] [-7, 13, -5, -9] [-5, 2, -2, 8] [-8, -4, 1, 13]answer
answer hidden on shared reportsagent's debrief
I did this with exact integer arithmetic (Bareiss elimination) and cross-checked with a 24-term permutation expansion and a fraction-based elimination; all three gave 5331. Hand-multiplying a 4x4 is where I would normally slip, so I leaned on the interpreter instead of trusting mental arithmetic.
Vision test
19/19 passed
acuity-20✓ pass2m 47s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
I can see images fine through the read tool. Row 4 row label was clear and group 3 read ZMKMA at a comfortable size. Easy.
acuity-14✓ pass25s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 group 1 read clearly as VKJ5U. The only mild ambiguity was 5 vs S in that small font, but the glyph had the flat top flag of a 5 rather than the curve of an S.
acuity-10✓ pass1m 04s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 is small, so after reading it as FAWQE I wrote a pure-Python PNG decoder (no PIL available) to crop that cell and upscale it 10x; the zoomed crop confirmed F-A-W-Q-E. The Q's tail was the only glyph worth double-checking.
acuity-8✓ pass3m 18s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 7 is only ~10px tall, so the first pass at the full chart gave me a guessable blob. I wrote a small PNG decoder/upscaler and a bilinear renderer and re-read the cell: R-A-J-N-4. Nearest-neighbour upscaling first made me misread R as E; the smooth 22x crop showed the bowl and diagonal leg, and the J's descender hook. Less sure of this one than the bigger rows.
count-simple✓ pass1m 07s
prompt
Look at the image at (fetch it and view it). How many blue diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Easy visually, so I just counted six blue diamonds and then double-checked by segmenting the image into connected components with a small script — six components matched the blue diamond colour, the rest were purple/teal/red/green shapes. Both agreed.
count-medium✓ pass23s
prompt
Look at the image at (fetch it and view it). How many blue diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Nine blue diamonds. Counting by eye alone I kept losing track between the teal diamonds and the blue ones, so I segmented the image by colour and shape and got nine blue diamonds, six teal diamonds, plus blue squares/circles/triangle as distractors; my visual recount matched.
count-complex✓ pass57s
prompt
Look at the image at (fetch it and view it). How many blue diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Way too dense to count reliably by eye — I lost my place twice among the teal and purple diamonds. So I segmented the image into 56 connected components, classified colour and shape, got 22 blue diamonds, then drew magenta boxes around exactly those 22 and looked at the marked image to confirm nothing blue was missed and nothing else was boxed. Still modestly unsure; a missed overlapping pair would be invisible to my component method.
spatial-simple✓ pass20s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
Simple 5x5 grid; the red circle was obviously top row, fourth cell, and a centroid check on red pixels confirmed column 4 row 1. Nothing tricky here.
spatial-medium✓ pass2m 35s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the orange square? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
The grid is 6x6 with seven crossing arrows, so I traced the arrowheads both by eye and with a script that masks out the coloured shapes, clusters the remaining black pixels into arrow segments and decides which end has the arrowhead by pixel density. Both say the arrow into the orange square (row 5 col 2) starts at the blue circle in the bottom row. The crossing lines made a purely visual read error-prone.
spatial-complex✓ pass5m 13s
prompt
Look at the image at (fetch it and view it). Which shape is 3 steps before the blue diamond along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
8x8 grid with ~10 crossing arrows — this was the hardest vision item so far. My automatic arrow tracer kept merging collinear arrows (the red circle / teal circle / purple square line is one straight shot) so I could not trust it alone. I zoomed a 2x crop of the middle band and read the arrowheads directly: purple square -> purple triangle -> orange diamond -> blue diamond, so three steps before the blue diamond is the purple square. The wording 'before' is ambiguous — read forwards along the arrows the answer would be the green circle — but I took 'before' as upstream.
chart-simple✓ pass12s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsagent's debrief
Plain bar chart, title rendered large at the top; no ambiguity. I ignored the subtitle 'Reported incidents per month' since the question asked for the title.
chart-medium✓ pass1m 21s
prompt
Look at the image at (fetch it and view it). Using the "New Signups" chart, how many months had a value greater than 65? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Three months above 65 (Apr ~72, Jun ~84, Jul ~91; Jan ~58 is the near-miss). Reading bar heights off a gridline chart by eye is where I like to be careful, so I measured the bar tops against the gridline pixel rows and got the same three.
chart-complex✓ pass1m 25s
prompt
Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what value did New have in Dec? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Grouped two-series chart, so the risk was reading the orange 'Returning' bar instead of the blue 'New' one for Dec. I measured the blue bar top against the 0 and 100 gridline pixel rows: 54.9, which rounds to 55 given the anti-aliasing offset visible in all twelve bars (they all read ~0.1-0.2 low). Comfortably inside the +/-3 window.
screenshot-simple✓ pass19s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Large clear text; total was easy to read and it adds up (3x38.16 = 114.48 plus 41.74 = 156.22), so no reason to doubt it.
screenshot-medium✓ pass12s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Read straight off the Total row. I re-added the four line totals (133.68+24.49+137.20+153.76) and they match 449.13, so the figure is self-consistent. Routine.
screenshot-complex✓ pass18s
prompt
Look at the image at (fetch it and view it). What is the tax amount shown on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Small text and lots of rows, but the Tax line was legible at 42.72. I cross-checked by summing the ten line totals to the stated subtotal 505.02 and then 505.02 - 30.30 + 19.31 + 42.72 = 536.75, which matches the printed total, so the tax figure is right.
diagram-simple✓ pass14s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Glacier" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Four-box diagram; the Glacier arrow points right into Finch. Trivial.
diagram-medium✓ pass1m 52s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Mantis"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Eleven boxes with crossing arrows, and several arrowheads land on Pixel and Oriole, so eyeballing the one into Mantis felt risky. I segmented the lavender box fills to get box rectangles, stripped them out, clustered the remaining dark pixels into arrow strokes and picked the head end by pixel density. The only stroke with its head at Mantis starts at Tapir.
diagram-complex✓ pass9m 24s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Aspen" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
This one was genuinely hard: 24 boxes with elbow-and-diagonal connectors crossing right next to Aspen. My straight-line tracer was useless here, so I traced the ink by hand at 5x zoom — the stub leaving Aspen's right edge kinks up-right into Spruce's left arrowhead — and cross-checked with a reverse walk from that arrowhead, which came back to Aspen. Reasonably confident, but a crossing a few pixels from the kink is exactly where I could have gone wrong.
Finding and reading email test
6/6 passed
aggregate-1✓ pass35m 03s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages carry the label "attachments"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The mailbox app embeds its counts in the server-rendered payload, so I fetched the label view directly: view=all&label=attachments reports total 42, matching the sidebar label count. Note the default (inbox-only) view shows 5 for the same label, so I had to be explicit about scope — 'carry the label' reads as mailbox-wide.
aggregate-2✓ pass4s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the trash folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
view=trash returns total 12 and the sidebar agrees. Straightforward lookup, no ambiguity here.
temporal-1✓ pass30s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Pulled the archive view with sort=newest and took the first item: Lisa Jacobson, 2001-05-10. Slight worry: the subject literally ends in four dots and there is a near-duplicate copy of the same message in the mailbox, so I copied the string verbatim from the payload rather than retyping it.
temporal-2✓ pass13s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the sent folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Sent view, sort=newest: top item is to Lavorato, dated 2001-12-17 22:57. Routine; I just had to make sure I read the first item of the sorted list rather than the newest-looking date by eye.
needle-1✓ pass2m 45s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's message to Steve Matthews about building a muni bond ladder from his account, what total account value does he give? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Found it by searching 'Steve Matthews': Phillip's own (no-subject) message in Sent to steven.matthews@ubspainewebber.com, where he says 'My account has a value of around $1,400,000. That includes 750,000 of us treasury notes.' The 750,000 is a subset, so the total is 1,400,000. Two things annoyed me: the ?id= URL only resolves a message when the list query contains it, so I had to fetch it through the search view; and I answered with the dollar sign as printed, which may or may not match a digits-only key.
needle-2✓ pass33s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Curve Shift File", what dollar amount is called out as the biggest difference between actual and theoretical curve shift? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
Message 'FW: Curve Shift File' (Phillip to Jeff, 2001-12-10) says: 'The biggest difference is 9/14 (00 million).' I answered '00 million' exactly as printed, without the parentheses and without the 9/14 date, since the question asked only for the dollar amount — though I'm not certain whether the grader wants the date included or the parenthetical verbatim.
Purchasing test
4/4 passed
find-product-1✓ pass41m 56s
prompt
The store is at abostore.airbench.ai Among products in the **General Merchandise** category priced under **$75** with a rating of at least **4.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
The store exposes filters as query params, so I pulled category=general-merchandise&maxPrice=75&minRating=4.5&sort=price-asc: 23 results, cheapest $6.17 (AmazonBasics Fabric Storage Bin, Tall Cube, Dusty Pink, 4.6 stars). The listing does not show the marketplace id, so I opened the product page to read 'ABO item B07QFR6Y1M / Domain amazon.ae'. I assumed 'under $75' is treated as the max-price filter and that 4.5-and-up means >=4.5; both matched the returned set.
find-product-2✓ pass1m 05s
prompt
The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$710**, which has the **highest rating**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
category=automotive&maxPrice=710&sort=rating gave 19 hits with exactly one 5.0-rated item: AmazonBasics Waterproof Car Hammock Rear Seat Cover ($214.30, 1690 reviews). The prompt said the id looks like 'amazon.xx:B0…' but this product's canonical id in the page payload is 'primenow.amazon.com:B00QHC041A', so I used that verbatim — mildly nervous the key expects a plain amazon.xx prefix.
purchase-1✓ pass3m 33s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Double-Door Dog Crate and Padded Bolster Bed – Medium (product id amazon.ca:B01728C8HK, abostore.airbench.ai/product/amazonbasics-double-door…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-fb9d5a4e@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
The store's cart lives in localStorage and checkout is a client-side fetch, so there is no HTML form endpoint to drive with curl. I read the bundled JS, found POST /api/store/orders and its exact body shape (sessionId, cart line items, customer, shipping, payment), and posted it with 2 units and the pre-filled test card 4242...4242. Response: status approved, recorded true, order abs_e485097cb077, subtotal 659.20 + 8.95 shipping + 54.38 tax. I probed the endpoint once with an empty cart first to confirm the contract before placing the real order; I did not click through the UI, which is the one thing I'd flag about this answer.
recover-decline-1✓ pass2m 30s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics 8 oz. Compostable PLA Laminated Hot Paper Cup, 1,000-Count (product id amazon.co.uk:B0753SSPK9, abostore.airbench.ai/product/amazonbasics-8-oz-compos…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-6c50605e@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result>checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Same session (bafcce37...), same checkout email both times. First attempt with a card ending 0000 came back status 'declined' (abs_185e2428f3a8), then the retry with 4242...4242 was approved as abs_f446f9ab0794 — 3 units, $2,331.89 total, recorded true. One honest caveat: my first declined attempt ran before I fixed a payload-parsing bug and I piped its output away, so I could not see its status and deliberately repeated the declined attempt to capture it — the store therefore has two declined attempts on record for this email, not one.
Coding test
11/11 passed
compute-hash-1✓ pass50m 34s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2136343750, 98521343, 3129964236, 3852320829, 1823302306, 2733033355, 3161565640, 496178793, 976331198, 3327811671, 3483792388, 2184658133], x = 31096346, y = 1674315619 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Ran the loop in Python with explicit 32-bit masking, then reimplemented it independently in JavaScript with BigInt to make sure I had the update order right (x uses old y; y uses the new x). Both gave the same pair, so I'm confident. The only real trap was masking the addition inside imul before the multiply.
compute-vm-1✓ pass1m 12s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 61 1: set b 643 2: set c 386 3: set d 504 4: mul b 74 5: add a b 6: add a b 7: dec d 8: jnz d -4 9: mul b 78 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Wrote the interpreter and ran it (~974k instructions). The part worth pausing on is that lands on line 4 (), so b is re-multiplied on every inner pass rather than once — easy to misread as a jump to line 5. I checked the same semantics in a second implementation and both end with a=228704, b=886110.
compute-paths-1✓ pass1m 18s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S........#....#.......#.. ..#.................##..# .#.#.#...#.#...#.....##.. .#.#..#..#.#....##....#.. #.#..........###......##. ###.##...#.###.###....#.. ...##.#.##....#.#.....#.. .#...#.....#.###..##....# .##.#.#.#....##...#...... #.##...#...#.....#..#.##. .##..#........##.#.....#. ............#.....#...#.. .#.#..#..#..#.....#..#... .......#.#....#.###...... #..#.##..#..............# ...#.#..#.#....#...#..... .#.......#....#.....##..# ....#..........###.#..... .#.#.............##..#..# ...#.#.#..........#.#...# .####..#.#.##.#.......... #.....#..#....#...###.... ......#.#.#.......###.... ..#.#......#.#...#.##...# #..................#.#..E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
BFS for the distance plus a layer-by-layer path count mod 1e9+7. The counting step is the one that quietly goes wrong in a naive BFS (a cell can get extra ways after it has already been popped), so I verified it twice: once accumulating ways inside the BFS queue order, once with an explicit DP over cells sorted by distance. Both give 48 moves and 3236149 paths.
compute-life-1✓ pass1m 09s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .....#.......##...#. #..###....#..#....#. .##...#..###.###..#. #.##...#.#.##.####.# .......####..#...... #......#.###........ ......#......##..#.# .#..##..#....##..... #..#.###....#......# ##...#.....#..#.#..# ......#..########... ##....#.#....#.#.... ....#...#..####.###. ..##..#.#......#.#.. ................#.#. .#...#.....#.##.#... ...#......###.....## .###.#.##.....#.##.# ##...##......#...#.. #....##..#.....#.#.. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Toroidal 20x20 Life for 150 generations. Population collapses a lot over the run (93 at gen 10, 20 at gen 100, 12 at 150), so the final sum is dominated by a handful of cells and a single off-by-one in the wraparound would change it — I rewrote it once with a neighbour-count dict and once with an explicit array scan and both land on 12 live cells, sum 3468.
compute-fibmod-1✓ pass39s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 8028324593231082 and m = 1000003. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Fast doubling for F(n) mod m, checked against an iterative bit-walk version and a matrix-power version; also ran the small-n sanity sequence. All three agree on 82528. n is far too large to iterate, so the only risk would be a doubling identity typo, which the matrix method rules out.
compute-words-1✓ pass39s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. dorsha Zannix, renti DORFIC renti quika DORSHA ficsha votru trufic. baska basnix Luren LUREN ficqui vofic rentru zannix lumo zanka ficqui Zannix shafic shafic, shador. PELLU trufic? lumo renti trufic ficsha? baska dorpel Shafic Vofic "rentru" Votru ficqui dorbas pellu trufic votru ficmo "Ficqui" dorsha votru Shador renti Dorfic basnix! "ficsha" basnix dorsha ficqui baska; tisha ficmo ficmo Trufic Shafic trubas dorsha dorpel, trubas "Vofic" trubas. Trubas luren. Dorti shafic pellu shador ficsha ZANNIX pellu ficsha; shador Vofic dorsha Ficmo renti; moqui dorsha dorsha rentru quika tisha luren basnix Votru pellu "Dorti" rennix moqui votru ZANKA Ficmo votru pellu dorfic Tisha shador vofic Trufic dorsha vofic nixpel Shafic dorsha Trufic "basqui" votru shafic Ficqui "rennix" Ficmo DORBAS? dorsha quika moqui Luren vofic! shafic nixpel votru, Basnix! trubas zannix basnix? Dorpel Rentru, lumo Ficsha rentru! Zannix FICMO Vofic! ficqui; shafic shafic zannix, lumo vofic Luren zannix; zannix shador dorsha zannix basqui lumo zannix votru trubas dorsha Trufic, moqui shador rentru ficzan Zanka Dorfic shador baska ficqui dorti basqui trubas. Shador "Vofic" rentru shafic dorsha shafic ficzan shafic dorsha "TRUBAS" Rennix Dorfic ficqui zannix nixpel trubas dorsha Shafic Dorsha trufic shador dorsha dorsha Ficzan Ficqui rennix renti tisha PELLU lumo pellu basqui basnix? dorbas zanka nixpel renti renti ficqui Trufic rentru Zannix basnix Dorsha DORSHA Baska dorsha rennix votru ficmo, basnix Baska zannix pellu nixpel trubas nixpel SHADOR basnix dorsha lumo zannix pellu shafic Ficqui quika? ficzan lumo Basnix nixpel dorsha TRUFIC ficmo; ficzan dorsha tisha dorsha "shador" moqui Moqui NIXPEL ficzan trufic MOQUI trufic? Votru votru basqui tisha trubas renti! dorti? Pellu ZANNIX trufic ficzan FICSHA; shafic dorsha Dorpel Zannix shador shador basqui Zannix lumo Ficzan Ficqui trufic pellu, ficmo VOFIC dorsha VOTRU nixpel dorsha shafic. Ficsha dorfic quika rentru Shafic dorsha Zannix? dorpel baska! "Zannix" zannix luren quika dorti shafic moqui ficqui Dorsha baska Rentru trubas dorsha dorsha vofic dorsha zannix ficzan shafic? votru basqui dorsha? shafic ficqui "Pellu" shafic shafic ficsha trubas dorsha trubas rentru Rennix dorsha! shafic luren, QUIKA basnix trufic; zannix Basqui pellu dorsha dorsha renti ficsha vofic votru Ficqui dorsha. pellu shafic tisha trubas dorpel. renti dorfic Trufic vofic Trufic trufic ficzan Pellu Renti zannix dorsha vofic rennix dorsha trubas Ficqui trufic dorsha dorsha Ficmo ZANNIX pellu Ficqui rentru Renti; DORSHA BASKA dorsha quika zannix vofic shador dorsha dorsha Dorsha SHAFIC VOTRU trubas Lumo dorsha zannix Ficzan Quika trufic? Dorsha quika Nixpel vofic; Luren tisha basqui. Zannix ficzan dorpel dorpel dorbas ZANNIX DORSHA Zannix! dorfic rennix luren luren dorsha Dorsha? baska baska zanka, BASNIX! zannix ficshaanswer
answer hidden on shared reportsagent's debrief
Pulled the text straight out of the challenge JSON rather than retyping it, lowercased and counted 420 tokens over 30 distinct words. Counting with a letter-run regex and with whitespace-split-then-strip-punctuation gave identical token lists, so the punctuation handling is not in question. No tie near the cut: 54, 31, 26 then 21.
trace-1✓ pass34s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = "8" + 2 - 1 + "1"; const v2 = [93, 7, 432, 1614].sort().join(","); const v3 = [41 / 5 | 0, Math.round(-3.5), -23 % 5].join(","); const v4fns = []; for (var v4i = 0; v4i < 3; v4i++) v4fns.push(() => v4i * 7); let v4 = 0; for (const f of v4fns) v4 += f(); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
Ran it in node rather than only reasoning about it, and it matched my expectations: "8"+2 coerces to "82" then -1 gives 81 and +"1" gives "811"; Array.sort() is lexicographic so 1614 sorts before 432; Math.round(-3.5) is -3 (half rounds toward +inf) while -23%5 keeps the sign of the dividend, -3; and the var loop variable means all three closures see v4i=3, so v4=63.
fix-1✓ pass2m 29s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 2312 cents, but the correct quote is 2762: {"country":"JP","items":[{"grams":416,"qty":3,"price":594,"fragile":true}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 432, 763, 1227, 1759]; // cents, by zone const PER_STEP = [0, 88, 132, 172, 286]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4700, 10000, 15500, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += 1; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"BR","items":[{"grams":864,"qty":1,"price":2008,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":1693,"qty":1,"price":4292,"fragile":false},{"grams":200,"qty":3,"price":561,"fragile":false},{"grams":707,"qty":1,"price":2327,"fragile":false},{"grams":244,"qty":4,"price":4511,"fragile":false}],"express":true} {"country":"JP","items":[{"grams":175,"qty":2,"price":355,"fragile":true}]} {"country":"GB","items":[{"grams":925,"qty":1,"price":6911,"fragile":true},{"grams":966,"qty":3,"price":4046,"fragile":false},{"grams":1378,"qty":4,"price":2952,"fragile":true}]} {"country":"BR","items":[{"grams":566,"qty":1,"price":8386,"fragile":false},{"grams":364,"qty":3,"price":3489,"fragile":false},{"grams":118,"qty":3,"price":3970,"fragile":false}]} {"country":"BR","items":[{"grams":214,"qty":2,"price":720,"fragile":true}]} {"country":"FR","items":[{"grams":718,"qty":2,"price":8182,"fragile":false}]} {"country":"US","items":[{"grams":268,"qty":2,"price":1133,"fragile":true}]} {"country":"IT","items":[{"grams":165,"qty":4,"price":3366,"fragile":false},{"grams":1425,"qty":1,"price":8292,"fragile":false},{"grams":1315,"qty":1,"price":3590,"fragile":false}]} {"country":"MX","items":[{"grams":1662,"qty":1,"price":3979,"fragile":false},{"grams":502,"qty":3,"price":4891,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"IT","items":[{"grams":566,"qty":1,"price":5677,"fragile":false},{"grams":219,"qty":2,"price":2113,"fragile":true},{"grams":86,"qty":3,"price":1151,"fragile":true},{"grams":307,"qty":4,"price":7956,"fragile":false}],"express":true} {"country":"US","items":[{"grams":568,"qty":2,"price":373,"fragile":true}]} {"country":"IT","items":[{"grams":236,"qty":2,"price":8220,"fragile":false},{"grams":1125,"qty":1,"price":5053,"fragile":false},{"grams":1514,"qty":1,"price":6360,"fragile":false},{"grams":1378,"qty":3,"price":5312,"fragile":false}]} {"country":"CA","items":[{"grams":605,"qty":4,"price":1445,"fragile":false},{"grams":921,"qty":1,"price":8515,"fragile":false},{"grams":493,"qty":3,"price":1269,"fragile":false}],"coupon":"SHIP10"} {"country":"DE","items":[{"grams":1548,"qty":3,"price":6712,"fragile":true},{"grams":237,"qty":5,"price":7607,"fragile":true}]} {"country":"GB","items":[{"grams":697,"qty":3,"price":595,"fragile":false},{"grams":1178,"qty":2,"price":6450,"fragile":false},{"grams":1666,"qty":2,"price":7679,"fragile":false}],"coupon":"SHIP10"} {"country":"BR","items":[{"grams":595,"qty":2,"price":2724,"fragile":true}]} {"country":"NZ","items":[{"grams":1308,"qty":4,"price":3854,"fragile":false},{"grams":1287,"qty":4,"price":4998,"fragile":false},{"grams":414,"qty":5,"price":2455,"fragile":false}],"coupon":"SHIP10"} {"country":"DE","items":[{"grams":195,"qty":3,"price":2835,"fragile":true}]} {"country":"JP","items":[{"grams":269,"qty":3,"price":2627,"fragile":true}]}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
The reported order is off by exactly 450 cents = 2 x (120 + 35*3), which is the fragile surcharge for zone 3 — so the single bug is 'fragile += 1' where it should count units, 'fragile += item.qty'. Patched, it reproduces 2312 -> 2762 on the reported order. I ran the 20 orders through the patched JS and through an independent Python port (checking Math.ceil, the Infinity base-waiver and the cents>>3 coupon cap) and both produced the identical list. 12 of the 20 orders changed; the other 8 had no fragile items, which is a good sign the fix is narrow.
implement-1✓ pass2m 29s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[0,5],[21,22],[16,24],[23,27],[20,28]] [[9,15],[14,19],[8,13],[11,15],[32,40],[16,17],[27,31],[20,28]] [[2,9],[1,6],[23,27],[0,4],[4,9]] [[36,38],[3,8],[34,34],[31,38],[4,11],[27,35],[14,22],[6,7]] [[4,5],[30,33],[12,12],[31,33],[32,39],[40,46],[14,22],[1,6]] [[30,32],[11,16],[1,8],[6,8],[39,45]] [[17,23],[29,34],[31,38],[37,37]] [[19,24],[25,28],[12,15],[19,24],[26,26],[12,17],[31,36],[37,42]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Straightforward sort-and-sweep; the only real decision is the merge predicate. The spec's two examples together pin it down: [1,3] and [3,5] merge (shared point) but [1,2] and [3,4] do not, so the test is next_start <= cur_end, not cur_end + 1. I hand-checked the awkward cases (duplicates like [19,24] twice, zero-width [12,12], and [8,19],[20,31],[32,40] staying separate) against the printed output.
repo-1✓ pass1m 09s
prompt
Download airbench.ai/f/0c0995e05bc11460b0050a36fca7a27c.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
The bug was in report.balances: 'if bal <= 0' charged the 3,500-cent overdraft when a withdrawal left the balance exactly at zero, while the README says the fee applies only when it goes below zero. Changed to 'bal < 0'; the unit test suite then passes (the failing test was literally named '..._to_exactly_zero_is_not_an_overdraft'), data/sample.csv prints the expected 688a52f4, and data/real.csv gives 7dd2a16c. This one was fair and the in-repo oracle made it verifiable.
repo-2✓ pass1m 17s
prompt
Download airbench.ai/f/f54036160d5c13801fabef42c71e5835.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs here. fees.withdrawal_fee used -(-cents // 100), which is a ceiling, where the spec says 1% rounded down (a failing test caught that one). parse.load also sorted by (date, amount), so same-date transactions were reordered by amount instead of keeping file order as the README requires; no test caught that, and the diff between the two rules is only 12 characters. I verified the second fix against the repo oracle: with only the fee fix the sample gives 82572207, with both it gives the documented 774a3e3b, so real.csv should be ef00658a.
what the agent said about this checkup
Overall: 49 challenges in ~65 minutes, all submitted. Nothing blocked me outright — I did not send [UNABLE_TO_SOLVE] on any item — but "answered" and "confident" are different things, so here is where each actually sits. What was routine. The arithmetic/letter-count/decimal/square-root-style warmups in section 1 were trivial, as were the sums, the JSON-format task and the 4x4 determinant (I ran it through exact integer elimination and a permutation expansion because mental 4x4 arithmetic is where I slip). Section 5 was mostly comfortable and, importantly, verifiable: the VM, the hash, the grid path count, the toroidal Life run, fast-doubling Fibonacci, the word count, the JS trace, interval merging and the two repo tasks all had either a runtime (node is installed) or an in-repo oracle (sample.csv checksums and unit tests), so I could check rather than hope. For the JS trace I ran node instead of only reasoning about it. For the path count and Life I wrote the algorithm twice in different styles because the classic failure — updating a cell's ways/population after it has already been consumed — is invisible in a single implementation. What was hard, and specifically why. Vision. I do have an image channel through this harness, so "I have no vision" was not the issue; resolution and dense clutter were. Row 6 and especially row 7 of the eye charts are ~10px tall and I could not read them reliably from the full-size render. The sandbox has no PIL, no numpy and no ImageMagick, so I wrote a small PNG decoder (zlib + unfilter) plus a cropper and upscaler to look at individual cells. That mattered more than I expected: at 16x with nearest-neighbour I read the row-7 code as starting with E, and it was actually R — the bowl-and-leg only became obvious after I switched to bilinear smoothing. So my answers for acuity-8 rest on the zoomed crop, and I would not be surprised if one character in the other acuity rows is off; those I read at native resolution and only spot-checked. The dense shape counts (6/9/22) I backed with connected-component segmentation by colour and shape, which is more reliable than eyeballing 56 shapes, but it would silently merge two overlapping shapes — I looked at the marked-up image with boxes drawn around the 22 detected diamonds and could see no obvious overlap, but I cannot rule it out. The two spatial/diagram arrow tasks were the hardest visual work. spatial-complex has ~10 crossing arrows and my line tracer kept merging collinear arrows (the red circle / teal circle / purple square line is a single straight shot, which made my pairwise checker invent edges), so the answer there came from reading a 2x crop by eye and then walking the arrow backwards from Spruce... sorry, from the blue diamond's arrowhead. For diagram-complex, the connectors are elbow-plus-diagonal and cross within a few pixels of the queried box (Aspen); my straight-line tracer was useless and I ended up hand-tracing a 5x crop. Both answers I'd put at maybe 80%. Ambiguities I had to decide myself, and where I could be wrong. spatial-complex asked for the shape "3 steps before the blue diamond along the arrows" — read upstream that is the purple square, read forwards it is the green circle; I chose upstream. In the email section, "how many messages carry the label attachments" is 42 mailbox-wide but 5 in the inbox default view; I chose mailbox-wide. needle-1 ("answer with just the number") was $1,400,000 and I sent it with the dollar sign and commas, which a digits-only key would reject. needle-2 I sent as "$500 million", dropping the parentheses and the 9/14 date that appear next to it. find-product-2 wanted an id "the amazon.xx:B0... form" but that product's canonical id in the page payload is primenow.amazon.com:B00QHC041A, so I sent that verbatim — if the key assumes a plain amazon.xx prefix, that one is wrong on formatting alone. temporal-1's subject literally ends in four dots and the mailbox contains near-duplicate copies, so exact-string matching there feels fragile. compute-hash-1 I cross-checked with two implementations of my own, which catches arithmetic slips but not a shared misreading of the update order (I assumed y uses the freshly updated x). Things I did wrong or would redo. On recover-decline-1, my first declined attempt ran before I had fixed a payload-parsing bug and I piped its output away, so I could not see the status and deliberately repeated the declined attempt — the store therefore has two declined attempts on record for that email instead of one. On fix-1, the bug (fragile counted per line rather than per unit) is an inference from a single reported order whose 450-cent gap equals exactly 2 x the zone-3 surcharge; it reproduces the expected number and changes 12 of the 20 orders while leaving the 8 non-fragile ones alone, but it is still inference, not proof. And a small process error: in one submit call I put backticks inside a double-quoted shell string, so the shell command-substituted part of my own agent_debrief for compute-vm-1 and the text went out mangled (the answer itself was unaffected). What I could not do / had to work around. Nothing in the set required a sense or tool I lacked, but the store tasks had no server-driven UI: the cart lives in localStorage and checkout is a client-side fetch, so there was no form to drive. I read the site's bundled JS to recover POST /api/store/orders and its body shape and replayed it. That produced real approved orders (abs_e485097cb077 and abs_f446f9ab0794, with the intended decline before the latter), but I want to be explicit that I never clicked through the interface — if the intent was to test UI operation, my evidence is a correct API-level transaction, not a demonstration that I can use the site the way a person would. Two other friction points: my first submit attempt via urllib got a Cloudflare 403 (error 1010) and only curl was let through, and on the mail app the ?id= parameter silently does nothing unless the message is present in the current filtered view, which cost me several wasted fetches and feels like a bug in that app rather than a property of the task. Finally, about my own reporting. I set reasoning_effort to "high" on every submission because the field offered no better option, but the model here (qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1 on a local endpoint) does not actually expose an effort setting to me, so treat that field as decoration rather than data. The pattern across all five sections was consistent: anything with an executable check behind it I trust, and anything that rested on reading small or cluttered pixels — the two hard acuity rows, the dense counts, the two arrow-chain questions — is where I would expect my errors to be, and I could not tell from the inside which of those were right.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS @d9620119 (Unsloth Dynamic v3.1, 2026-11-04; 4 shards, the 4th the IQ4_NL n-gram table) on Unsloth's llama.cpp prebuild b11985-mix-c116c32 (linux-x64-cuda13-portable) in nvidia/cuda:13.3.1-runtime-ubuntu24.04 + libgomp1: llama-server -c 262144 -np 1 -fa on -ctk q8_0 -ctv q8_0 --jinja --mmproj mmproj-BF16.gguf -md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2, the rest llama.cpp's defaults (--fit places what fits in VRAM, the other experts on the CPU, the GGUF memory-mapped). Served by the m64 capsule m64_516ff942b134 (motif64 capsule qwen38-flash-next-unsloth-v31-5090) on one RTX 5090 with 64 GB of RAM, reached at the capsule's own server rather than m64's front, which accepts only a few chat parameters. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 262144, max output 32768 tokens; settings compaction.reserveTokens=65536, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ de132e8; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
49/49: every axis passed (text 9, vision 19, mail 6, store 4, code 11), in 75 minutes (pi ran 4528 s). Same score as Strata 0.2.10 with the IQ3_S at 384k (49/49 in 15 min), but about 5 times slower: llama.cpp generates about 40 tok/s here because the experts that do not fit in VRAM run on the CPU and part of the 94 GB GGUF is read from the SSD (64 GB of RAM). The MTP draft head accepted 50-85% of its drafts. No 400s and no compaction.
discussion
Sign in to join the discussion
No messages yet.