Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-iq3_s-basalt
setup
- model type
- open model (local)
- hardware
- RTX5090
- harness
- pi
- model
- qwen3.8-flash-next-iq3_s-basalt
- VRAM
- 32 GB
- RAM
- 64 GB
started 2026-10-11 21:32 UTC
overall
Answered 49 of 49 challenges; 48 correct.
48 of 49 challenges passed
- 48 passed
- 1 failed
vitals
time
9m 42s
answered
100%
failed
2%
success
98%
systems
Math test
9/9 passed
letter-count-1✓ pass17s
prompt
How many times does the letter "z" appear in "zbaszaznzpel"? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Totally routine - I counted the z's by eye in zbaszaznzpol and then double-checked with a one-line count in Python. Only real risk is careless counting on a longer string.
decimal-compare-1✓ passbatched
prompt
Which decimal number is larger, 9.7 or 9.73? Answer with just the larger number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Easy comparison once I treated them as 9.70 vs 9.73. This one is a trap for models that compare digit strings, so I deliberately padded to equal decimal places before deciding.
arithmetic-1✓ passbatched
prompt
Compute step by step, left to right (no operator precedence): 37 - 1 / 6 * 7 / 2. Answer with just the final number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Straightforward but I had to override my usual precedence habit: strictly left to right it is 36, 6, 42, 21. I sanity-checked the intermediate values in Python rather than trusting the mental run.
unit-convert-1✓ pass2s
prompt
Convert 17 GB to MB. Now treat that resulting number as a fresh quantity of kg and convert it to g (1 kg = 1000 g). Answer with just the final integer number of g.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The arithmetic is trivial (17 -> 17000 -> 17000000) but the task is ambiguous about GB->MB: SI would give 17000 MB while binary (MiB) would give 17408 MB. Since the second hop is explicitly stated as a factor of 1000 I assumed the same decimal convention throughout and answered 17000000; if the grader used 1024 the expected answer would be 17408000.
format-json-1✓ passbatched
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "4857". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 4857. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Formatting exercise, easy. The only gotcha is that checksum has to be a bare JSON number, not a string, and the key order has to be answer then checksum - I built it with an ordered dict and checked 4+8+5+7=24.
math-add-1✓ passbatched
prompt
What is 16 + 9? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Trivial addition, 16+9=25. Nothing to struggle with; I just verified it to be sure I wasnt off by one.
math-sum-1✓ passbatched
prompt
Compute the exact sum of 137 + 359. Respond with just the integer.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Routine two-digit/three-digit addition (137+359=496). Verified in Python; this kind of task never gives me trouble.
math-expression-1✓ passbatched
prompt
Evaluate this expression exactly and respond with just the integer: ((18 + 14) * (14 - 12)) + (6 * 7) - 60answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Plain expression evaluation: (18+14)*(14-12)=64, plus 42, minus 60 = 46. I expanded it step by step and cross-checked with Python, so I am confident.
math-determinant-1✓ passbatched
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [11, 1, 7, 3] [8, 11, -9, 7] [7, -8, 8, 1] [-8, -9, -7, 13]answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The 4x4 determinant is the only one where hand arithmetic could plausibly go wrong, so I did not do it by hand: I computed it twice, once by fraction-exact Gaussian elimination and once by the full permutation/Laplace expansion, and both gave -13174. My first attempt actually printed -13173.999999999995 because I mixed in float division, which is exactly the kind of silent error this task is good at catching.
Vision test
18/19 passed
acuity-20✕ fail49s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 0
expected 8QBVG, got "8QBUG"
agent's debrief
Row 4 group 1 was large enough to read straight off the full chart, and I cropped and upscaled it to confirm 8QBUG (checking 8 vs B and G vs C). Comfortable; the smaller acuity images should be much harder.
acuity-14✓ pass4s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Row 5 group 2 read as 3QFHR. At full-image size the glyphs were only a few pixels tall so Q vs O and R vs P were genuinely ambiguous; cropping and upscaling 5x made it clear. I would not have trusted the un-zoomed view.
acuity-10✓ pass6s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Row 6 group 2. The full chart rendered too small for me to be sure of 8 vs B and M vs N, so I cropped that single group and upscaled it about 6x; it reads Z8MXC. Zooming is doing most of the work here, not my raw acuity.
acuity-8✓ pass5s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Row 7 group 2 was the hardest - the glyphs are only about 8 pixels tall and unreadable in the full-size view. Cropping to the single group and blowing it up 8x (both interpolated and nearest-neighbour) gave 9VMX4, though the 9 and the trailing 4 are barely resolved and I would not be surprised if the truth were g/8 or K.
count-simple✓ pass7s
prompt
Look at the image at (fetch it and view it). How many teal circles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Easy. I counted the teal circles visually (3) and confirmed it programmatically by flood-filling the exact teal RGB into connected components - three blobs of identical area. Having the pixel-level tool makes counting tasks about colour naming rather than counting.
count-medium✓ pass5s
prompt
Look at the image at (fetch it and view it). How many green circles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The trap here is that green also appears as diamonds and triangles, so counting by colour alone would overshoot. I counted green connected components in code and separated circles from diamonds by pixel area (9380 vs 5940 px), then checked the picture by eye: 9 green circles.
count-complex✓ pass10s
prompt
Look at the image at (fetch it and view it). How many orange circles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Honestly this one is not countable by eye - 35 orange shapes scattered over the canvas, with red dots that look almost the same at a glance. I fell back on pixel analysis: flood-filled the exact orange RGB (242,106,34) and classified each blob by fill ratio, giving 22 circles, 6 squares, 4 diamonds and 3 triangles. I am more confident in the code than in my own eyes here, though orange-vs-red is a judgement call the grader might make differently.
spatial-simple✓ pass4s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Trivial grid lookup. I eyeballed it as row 1 column 4 and then confirmed by finding the only red blob in the pixel data (centre about x=852,y=147, which lands in column 4 of row 1). No difficulty.
spatial-medium✓ pass13s
prompt
Look at the image at (fetch it and view it). Which shape does the arrow from the orange diamond lead to? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The picture has two arrows meeting at the orange diamond (one in from the blue square, one out), so I did not want to guess direction from the thumbnails. I extracted the dark line segments from the pixels and measured where the thicker arrowhead sits at each end: the segment touching the orange diamond has its head at the purple circle, so orange diamond -> purple circle. Visually plausible too, and the pixel trace made the direction certain.
spatial-complex✓ pass44s
prompt
Look at the image at (fetch it and view it). Which shape is 2 steps before the purple square along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
This was the hardest vision task so far - 65 shapes and 14 crossing arrows, and the arrowheads are tiny. My first automated attempt at tracing segments got the direction wrong on one edge (segments break where arrows cross, and my endpoint-density heuristic was fooled), so I fell back to zoomed crops: the arrow into the purple square comes from the green triangle, and the arrow into that green triangle comes from the blue square. I am reasonably but not fully confident; the arrowhead at the green triangle is clear, the identity of its tail shape is inferred from where the line exits the blue square.
chart-simple✓ pass7s
prompt
Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what value did May have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Reading a bar off an axis by eye is fuzzy, so I measured it: baseline at y=619, the 50 gridline at y=119.5, May bar top at y=190, which works out to 42.9. The other bars came out at clean values (27, 39, 29, 45) so I trust the calibration and say 43.
chart-medium✓ pass3s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Easy - the title is large and crisp. The only mild ambiguity is that there is also a subtitle, "Unique users per month, in thousands", but the bold heading is clearly the title, so I gave just that.
chart-complex✓ pass8s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, how many months did Paid have a value greater than 46? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Counting twelve orange bars against an invisible 46 line by eye is error-prone, so I measured each orange bar top in pixels and calibrated with two gridlines (100 at y=119.5, 75 at y=259.5). Paid comes out about 38,20,26,69,22,70,52,35,67,83,87,83 - seven months above 46. The closest call was July at ~52, still clearly over.
screenshot-simple✓ pass3s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Easy read of the Total line. I also added the line items (38.38 + 61.76 + 35.68) and they sum exactly to 135.82, so the screenshot is internally consistent and I am confident.
screenshot-medium✓ pass3s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Another straightforward Total read; the digits were crisp. I re-added the line totals (93.94 + 110.28 + 75.20 = 279.42) and they match, so nothing looked off in the data.
screenshot-complex✓ pass4s
prompt
Look at the image at (fetch it and view it). What is the tax amount shown on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The rows are small but legible. The Tax line reads 26.55 and I did not need to trust my reading blindly: subtotal 343.07 equals the sum of the nine line totals, and 343.07 - 48.03 + 18.41 + 26.55 = 340.00, exactly the printed total, which confirms I read the tax right.
diagram-simple✓ pass4s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Beacon" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Simple flowchart: Copper feeds Quiver, Beacon and Lemur, and the only arrow leaving Beacon goes straight to Rowan. Easy once I upscaled the 568x288 original so the labels were legible.
diagram-medium✓ pass7s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Aspen"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The graph has crossing edges near the bottom, and Eagle has two outgoing arrows (one to Aspen, one to Piano), so I zoomed right in on the top edge of the Aspen box to count arrowheads: exactly one enters Aspen, coming down from Eagle. The second line just passes over the top of the box on its way to Piano.
diagram-complex✓ pass34s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Hazel" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Hardest of the diagram set - 24 boxes with long, bending, crossing polylines. I wrote a small pixel walker that leaves Hazel along each dark line touching its border; the only line that leaves Hazel without an arrowhead at the Hazel end runs left and drops into an arrowhead on Zircons right edge. I confirmed it with a zoomed crop showing the arrowhead on Zircon and a plain (tail) end at Hazel. Reasonably confident, but the other two lines at Hazirs left edge are incoming arrows, which made this easy to get backwards.
Finding and reading email test
6/6 passed
aggregate-1✓ pass4m 50s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The mailbox is a Next.js app with no JSON API, so I scraped the embedded flight payload and counted sent messages with hasAttachments:true across all 3 pages - 17. I cross-checked with the built-in label filter view=sent&label=attachments, which also reports 17, so I feel good about this one.
aggregate-2✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the archive folder? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Same scraping approach as the previous one: I pulled all 92 archive messages across pages and counted unread:true, giving 41. It also reconciles with the global unread count of 50 minus the 9 unread in the inbox, which is a reassuring consistency check.
temporal-1✓ pass10s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
I sorted the archive newest-first and took the top row (2001-05-10), then opened the message to copy the subject verbatim, including the four trailing dots which are part of the real subject string rather than an ellipsis I added. Slight risk that the grader trims punctuation, but I answered exactly as shown.
temporal-2✓ pass7s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the inbox folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
I used the mailbox sort=oldest on the inbox and took the first row (2001-04-24), then opened the message to copy the subject exactly, including the odd "DRAFT- " prefix with no space after the hyphen. Straightforward.
needle-1✓ pass1m 17s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
This one took real work: the built-in search only indexed two of the three Huntley messages, so the thread about the lot survey never came up for the terms Huntley, survey, lot or fax. I ended up downloading the full body of every message in the mailbox (265 listing URLs, since a message only renders when it is in the current listing page) and grepping locally, which turned up RE: Huntley followup question in the Trash folder. Bob writes please send it to my fax at 281-858-1127, distinct from his main number 281-858-0000 in the signature, so I gave the fax line only.
needle-2✓ pass6s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Renee's reply to Phillip about the Deferred Phantom Stock Units payout, how many shares does she say are reflected on the 9/30/01 statement? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Found it in Renee Ratcliffs reply to Phillip: she writes that they will distribute the shares reflected on his 9/30/01 statement, 6,606 shares plus cash for fractional shares. I gave the bare number 6606 rather than the comma-formatted 6,606 as it appeared, since the prompt asked for just the number; that formatting choice is the only thing I am unsure about.
Purchasing test
4/4 passed
find-product-1✓ pass6m 56s
prompt
The store is at abostore.airbench.ai Among products in the **Tools & Hardware** category priced under **$300** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The store exposes server-side filters, so this was mostly discovery: I had to find that category, minRating and sort go in the query string. With category=tools-and-hardware, minRating=3.5 and sort=price-asc the first hit was AmazonBasics diagonal cutters at $9.46 (rating 4.8), well under the $300 cap, and I read the amazon.com:B07TWFDN96 id off the product page payload since the listing only shows the slug.
find-product-2✓ pass28s
prompt
The store is at abostore.airbench.ai Among products in the **Sports & Outdoors** category priced at or above **$650** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Harder because the store rating filter only offers 3.5/4/4.5, not 3.6, so I scraped all 35 pages of the Sports and Outdoors category (854 items) and filtered myself: 288 items at $650 or more, and the cheapest with rating at least 3.6 is a Helly Hansen jacket at $652.02 rated 4.4. The items just below it at $654-$658 are rated 3.5 or 3.4 so they are correctly excluded. I then had to read the real product id off the item page because the domain is amazon.com.au, not amazon.com - the slug alone would have given the wrong answer.
purchase-1✓ pass36s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics Pro-Style Spring Sprayer Kitchen Faucet, Polished Chrome (product id amazon.ca:B07F2C7S8Y, abostore.airbench.ai/product/amazonbasics-pro-style-s…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-5095e151@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader · score 1
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
No browser was available, so I had to reverse-engineer the checkout: the cart lives in localStorage and the form posts JSON to /api/store/orders, which I found by reading the minified Next.js chunk for the checkout page. I sent the same payload the page would send (3 units at $255.16, ground shipping, the given email, card ending 4242) and got back approved with order id abs_12d09a96d1c7. Slightly uneasy that I bypassed the UI, but the server recorded a real order with subtotal 765.48 and total 837.58.
recover-decline-1✓ pass9s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of Franklin Tailored Men's Box Silk Bowtie, Black/Silver (product id amazon.ca:B016PYXDL2, abostore.airbench.ai/product/franklin-tailored-men-s-…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-2de754bd@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader · score 1
trace:checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Same reverse-engineered checkout endpoint as the previous task, so the mechanics were easy by then. I made the first attempt with a card ending 0000 and got status declined (order abs_a5cca05758c2), then retried with a different valid card and got approved (abs_6edf74944e11), keeping one session id and the same checkout email for both, as asked.
Coding test
11/11 passed
compute-hash-1✓ pass8m 17s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [1806812123, 729302616, 3091554361, 1454918094, 3675261863, 4254419860, 586661285, 169117994, 3134140851, 4193768464, 3767256913, 1998909894], x = 2985058815, y = 1316092876 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Rote but fiddly: the only traps were modulo-2^32 discipline and the order of updates, since the new x feeds into the y line and the new y into the third x line. I wrote it in Python with explicit % 2**32 after every multiply and XOR, and masked y after the XOR because Python ints are unbounded. Nothing felt uncertain beyond a possible off-by-one in which x each line uses.
compute-vm-1✓ pass6s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 753 1: set b 351 2: set c 331 3: set d 512 4: sub a 94 5: mul a 20 6: mul a 23 7: dec d 8: jnz d -4 9: add a b 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
I implemented the little VM literally in Python rather than reasoning about the nested loops, because the negative modulo behaviour of sub/mul is exactly where a hand calculation goes wrong. It ran about 849k steps and ended with a=340003 and c and d both 0, which matches the two loop counters, so the control flow looks right.
compute-paths-1✓ pass6s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#...###......##..##..#.# ...#..#........#..#...... #....#.#.#..#.##.....#..# ....#..#.#...#..##...#... ........#.#......##...#.. #...##.##.###............ .###.#....#.....#.###.... #................#.#..... ..#.#.#..##.#.#......#... .#...#..#..#...##...#...# ..#....##.##..#.....##... #......##....#......#...# #..##.......###.....##.## ....#..#.##..##..###..... #..#....#...#....#...#.## ..#....#...###.........#. .#...#....#.#..#.#.#...#. ..#......#....#...#..#..# ....#...........##.#...#. ##.............#.#...#.#. .....##.#.#..#.....#..... ..##.#..#..#......#...#.. ..#.#.........#..#....#.. ..#..#.#....#...#..##.... #.#.......#.##......##..E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Standard BFS with a path-count accumulator, which is routine for me. The one thing worth being careful about is that a cells count is only final once every cell one step closer has been popped, which BFS ordering guarantees; I checked the grid was read in as 25x25 so no rows were clipped.
compute-life-1✓ pass7s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .#.#.#.###...#####.. ....#.....#..#.#...# ##...##...#.##..#.#. ##.#..#.#.##........ #..###...#......##.. ..##.#..##.....#.#.. #..#.......#...#..## ......###.....##.... #..##...#...##...... #...........##..#... #....#.#.....#...... #...###...#..##..### ##...##.#.#.#....#.# .#...#..#.##.#...... #.####..#.#..#...... ..##......##.###..## .........#.......... .#.....##..#..#..... ##......##...#.....# .#..#..###..###.#.#. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Routine toroidal Game of Life; I kept it as a set of live cells with a neighbour counter, which is the safe way to get the wrap-around right. My first run crashed on a silly set-comprehension unpacking mistake, so I fixed it and asserted the grid is really 20 rows of 20 before running 150 generations.
compute-fibmod-1✓ pass10s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 7056069921562448 and m = 2750159. Respond with just the integer.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Fast doubling modulo m is straightforward, but I wanted an independent check so I also factored m (it is prime), worked out the Pisano period 2750158 and recomputed F(n mod period) - same answer, 1044937. My first attempt had a tuple-precedence bug in the return statement that crashed it, which is worth admitting: I only caught it because the error was loud rather than silent.
compute-words-1✓ pass6s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. luvo? quiti kabas Luren; quipel? mobas peltru vodor kafic renqui, luren shador! vodor. Vodor, zanvo vobas Quipel dorlu ficlu basti pelpel peltru zanvo Vodor truren! Monix Tific quipel pelpel! Shaka Ficqui nixzan trumo quipel tific trumo! mobas "Vodor" vodor kabas nixzan renqui; truren MOBAS vodor shabas kabas DORPEL. Truren Trumo nixzan quipel; tific luvo; peltru basdor "kafic" dorlu Trumo! luvo, basdor pelpel luren luvo shaka zanvo. Nixzan vodor "Renqui" vodor dorpel shador luvo luvo "mobas" peltru pelpel dorpel kafic quipel zanvo ficnix "vodor" Vodor! Shabas. dorsha pelpel kafic peltru Ficqui Dorlu dorpel vodor mobas. vodor vodor. Mobas. "mobas" renqui. ficqui renqui monix mobas ficqui nixzan, Vodor tific. Vodor zanvo FICLU pelti zanvo, tific dorpel truren dorpel trumo tific Vodor peltru kafic Luren Ficnix DORLU "luren" Trumo Basti luren VODOR dorpel luvo quipel Dorlu ficqui Quipel Kabas dorpel quiti mobas quiti, pelti vodor shabas ficqui? quipel vodor ficnix basdor; zanvo kafic Tific vodor trumo; nixzan quiti vodor Ficlu "Vodor" truren. Dorlu truren dorlu dorlu Quiti ficlu. Luren basti peltru renqui ficqui Mobas. vodor Zanvo ficqui Shabas basti vodor tific Ficnix. mobas Pelti nixzan zanvo tific Dorsha Quiti shador luvo quiti tific vodor "vodor" dorsha pelti FICQUI zanvo dorpel tific vodor kafic Tific! ficnix nixzan shabas shaka ficqui shador Basti dorlu shador Peltru Basti? dorlu luvo tific shador renqui vodor ficqui Kabas renqui ficnix? kabas "Luren" Shabas peltru dorlu vodor shaka nixzan peltru luren FICQUI ficqui basdor Basdor dorlu Vodor; ficqui luvo vodor peltru Quipel. ficnix ficqui vodor FICQUI basti "Trumo" Ficqui tific Tific DORPEL nixzan Shador, tific ficqui peltru Monix Tific? ficnix Basdor kabas "quiti" Ficnix Peltru QUIPEL Basti tific kabas "basti" kafic Dorlu! quipel quipel. kafic ficqui Luvo vodor; truren Trumo ficqui peltru basti Peltru "dorlu" luvo! ficqui; quipel shaka dorpel; Monix "Mobas" luvo ficqui SHAKA nixzan nixzan vodor basdor kafic quiti dorlu pelti quipel quipel Vodor zanvo mobas vodor luvo pelpel Peltru. vodor, vodor kabas. vodor luvo Renqui peltru Dorpel dorpel tific Pelpel, luvo shabas; "dorlu" dorlu dorlu nixzan pelpel FICQUI luvo "Vodor" luvo Basti vobas ficqui LUVO shador ficnix trumo ficnix vodor "kabas" Shabas; nixzan peltru trumo Kafic quiti "kabas" tific Quiti quipel luvo monix vodor ficnix renqui shador ficqui! ficqui kabas vodor mobas Tific quipel Ficnix vodor quiti ficnix shador Pelti trumo quipel vobas peltru SHADOR quiti quiti ficlu trumo. vodor truren zanvo vodor "BASTI" shador truren ficqui "tific" tific renqui ficqui Mobas peltru quipel ficlu quiti dorlu renqui dorsha quipel dorpel Ficqui vodor basdor. renqui Kabas KABAS "dorlu" luren truren? ficqui peltru? mobas Vodor shaka peltianswer
answer hidden on shared reportsgrader · score 1
agent's debrief
Straightforward counting; I extracted words with a letters-only regex after lowercasing, which is equivalent to stripping attached punctuation here (420 tokens, 30 distinct). Closest tie risk was 4th place, peltru and quipel both on 21, but the top three are clear.
trace-1✓ pass4s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = (0.1 * 4 + 0.2 * 4 === 0.3 * 4) ? "equal" : "different"; const v2 = ["20" < "3", null >= 0, null == 0].map(Number).join(""); const v3arr = [4, 9]; v3arr[5] = 5; const v3 = v3arr.length + ":" + v3arr.filter(() => true).length; const v4 = [14 / 3 | 0, Math.round(-8.5), -97 % 5].join(","); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsgrader · score 1
agent's debrief
I had node available so I just ran the snippet instead of reasoning about float comparison, sparse arrays and the null coercion rules in my head. The interesting traps (0.1*4+0.2*4 not equalling 0.3*4, Math.round(-8.5) rounding toward +infinity, filter skipping array holes) all showed up in the actual output and matched what I expected.
fix-1✓ pass18s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1522 cents, but the correct quote is 670: {"country":"GB","items":[{"grams":1173,"qty":1,"price":11300,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 422, 852, 1216, 1681]; // cents, by zone const PER_STEP = [0, 72, 134, 179, 253]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4400, 11300, 15300, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value <= FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"NZ","items":[{"grams":668,"qty":1,"price":1702,"fragile":false},{"grams":1020,"qty":1,"price":4764,"fragile":true},{"grams":1558,"qty":2,"price":2434,"fragile":false},{"grams":903,"qty":1,"price":6105,"fragile":false}]} {"country":"IT","items":[{"grams":140,"qty":1,"price":4400,"fragile":false}]} {"country":"US","items":[{"grams":429,"qty":1,"price":5444,"fragile":false},{"grams":707,"qty":5,"price":4426,"fragile":false}],"express":true} {"country":"BR","items":[{"grams":1537,"qty":1,"price":15300,"fragile":false}]} {"country":"US","items":[{"grams":1384,"qty":4,"price":8074,"fragile":false},{"grams":309,"qty":4,"price":3211,"fragile":false},{"grams":1165,"qty":4,"price":3239,"fragile":false}]} {"country":"DE","items":[{"grams":310,"qty":1,"price":4147,"fragile":false},{"grams":769,"qty":3,"price":4997,"fragile":true},{"grams":886,"qty":1,"price":1291,"fragile":false},{"grams":1050,"qty":1,"price":8887,"fragile":true}]} {"country":"BR","items":[{"grams":1977,"qty":1,"price":15300,"fragile":false}]} {"country":"US","items":[{"grams":814,"qty":1,"price":11300,"fragile":false}]} {"country":"JP","items":[{"grams":1210,"qty":3,"price":2239,"fragile":false}]} {"country":"ES","items":[{"grams":1565,"qty":1,"price":4400,"fragile":false}]} {"country":"CA","items":[{"grams":826,"qty":1,"price":7819,"fragile":false}]} {"country":"ES","items":[{"grams":406,"qty":2,"price":5134,"fragile":false},{"grams":863,"qty":2,"price":8330,"fragile":false},{"grams":1046,"qty":5,"price":3264,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":233,"qty":3,"price":6264,"fragile":true},{"grams":267,"qty":1,"price":6784,"fragile":false},{"grams":1527,"qty":5,"price":2664,"fragile":false},{"grams":1255,"qty":1,"price":8196,"fragile":false}]} {"country":"ES","items":[{"grams":544,"qty":4,"price":519,"fragile":false},{"grams":1659,"qty":4,"price":4899,"fragile":false},{"grams":858,"qty":1,"price":4146,"fragile":false}],"express":true} {"country":"GB","items":[{"grams":809,"qty":1,"price":11300,"fragile":false}]} {"country":"NZ","items":[{"grams":388,"qty":1,"price":8649,"fragile":true},{"grams":102,"qty":4,"price":6079,"fragile":true},{"grams":1733,"qty":5,"price":7069,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"BR","items":[{"grams":824,"qty":1,"price":7396,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"FR","items":[{"grams":603,"qty":1,"price":8007,"fragile":false}],"coupon":"SHIP10"} {"country":"JP","items":[{"grams":452,"qty":1,"price":15300,"fragile":false}]} {"country":"NZ","items":[{"grams":145,"qty":1,"price":3985,"fragile":false},{"grams":1178,"qty":4,"price":5986,"fragile":true},{"grams":1528,"qty":1,"price":8318,"fragile":false}]}answer
answer hidden on shared reportsgrader · score 1
20/20 outputs match
agent's debrief
The failing order showed the base-fee condition was inverted: FREE_BASE_OVER is meant to WAIVE the base fee, so the base should be added only when value is strictly below it. Porting quote to Python and flipping <= to < reproduces 670 for the reported order. I am genuinely unsure about one thing though: the || order.express clause in that same condition, which makes every express order pay the base fee regardless of value. I treated it as intended behaviour (one bug means one operator flip), but if it was meant to be removed then 3 of my 20 numbers (the express ones at 4494, 4089 and 4845) would instead be 3216, 3456 and 4212.
implement-1✓ pass11s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[39,46],[16,22],[38,38],[37,45]] [[10,15],[18,22],[7,8],[31,32],[18,20],[12,13]] [[0,0],[37,41],[6,12],[17,24],[35,42],[39,40],[30,37]] [[21,24],[15,15],[36,39]] [[29,36],[15,18],[2,9]] [[38,46],[26,28],[40,42],[12,19],[18,26]] [[0,5],[36,42],[29,30],[36,43],[31,38],[2,6],[12,18]] [[33,40],[14,22],[29,37],[14,16],[27,28],[4,11]]answer
answer hidden on shared reportsgrader · score 1
12/12 lines match
agent's debrief
Interval merging is trivial, but I nearly got it wrong on the definition of touching: my first run merged [1,2] with [3,4], which the spec explicitly says must stay apart. The examples caught it, so the merge test is strict overlap (next.start <= current.end), not adjacency. Worth flagging that the wording touching is misleading given that example.
repo-1✓ pass9s
prompt
Download airbench.ai/f/44989bc76d2087af35df2c99ed4a4433.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The bug jumped out once I ran the bundled tests: withdrawal_fee used a ceiling trick, -(-cents // 100), where the README and the failing test want 1 percent rounded down. One line change and the whole suite passes, and data/sample.csv then prints the documented 213b5269 which is a good confirmation before trusting real.csv. Nice task, the self-verifying sample checksum makes it hard to fool yourself.
repo-2✓ pass8s
prompt
Download airbench.ai/f/c3f4b3103823320732e19fc97fcaf3a2.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Two bugs here and only one was covered by tests: the withdrawal-fee threshold used > instead of >=, which the unit test flagged, and load() sorted transactions by the raw DD/MM/YYYY string instead of the date_key tuple, which no test noticed but which scrambles the order the README promises. I fixed both and the sample file then printed the documented f28bf8a8, so I am confident before trusting real.csv. I did briefly worry there was a third bug since the prompt said bugs, but the documented sample checksum matching is strong evidence there were only two.
what the agent said about this checkup
Overall the checkup split cleanly into what I am good at and what I am not. Routine and fast: the math section (all nine, including the 4x4 determinant, which I computed twice - fraction-exact elimination and the full Leibniz expansion - because my first attempt silently printed -13173.999999999995 from float division), the coding/compute section (hash loop, tiny VM, BFS shortest-path counting, toroidal Game of Life, fast-doubling Fibonacci, word frequencies, interval merging, both repo bug hunts), and reading numbers off charts and order summaries where I could cross-check the arithmetic (the order summary added up exactly to $340.00, which confirmed I had read the tax line correctly). What was hard was anything that needed real visual resolution or spatial tracing. The eye charts were only legible after I cropped each group out of the PNG and upscaled it 5-8x; acuity-20 was fine at full size but acuity-10 and especially acuity-8 were guesswork without zooming, and I answered 9VMX4 for acuity-8 knowing the 9 and the trailing 4 were barely resolved - that is probably my weakest confident answer in the whole run. spatial-complex was worse: 65 shapes and 14 crossing arrows. I wrote a pixel walker to follow the arrows and it got the direction of one edge backwards, because segments break where one arrow's white halo crosses another and my arrowhead-density heuristic was fooled. I had to fall back to zoomed crops and read the chain by eye (blue square -> green triangle -> purple square). I would put that one at maybe 70%. Counting was hard in a different way: count-complex had 35 orange shapes among lookalike red ones and I simply cannot count that reliably by eye, so I flood-filled the exact RGB and classified blobs by fill ratio (22 circles, 6 squares, 4 diamonds, 3 triangles). Same for find-product-2, where the store's rating filter only offered 3.5/4/4.5 and I had to scrape all 854 Sports & Outdoors items and filter myself. That is an uncomfortable place to be: I am trusting code, and if my colour-name or shape-classification rule is wrong, my answer is confidently wrong. The email section had the most friction. The mailbox is a server-rendered Next.js app with no API, so I scraped the embedded flight payload, and worse, its own search index missed the message I needed: searching Huntley, survey, lot and fax never surfaced the "RE: Huntley followup question" thread, which was sitting in Trash. I ended up downloading all 265 message bodies (a message only renders when it is in the current listing page) and grepping locally. Two things looked broken or unfair there: the search index is incomplete, and some folder counts disagreed with the number of listed items (inbox said 24, listed 23; all mail said 178, listed 177) - one message was simply unreachable from the listing. That cost more time than the needle questions themselves. The purchasing test needed a browser and I had none. I found the checkout payload shape by reading the minified JS, then POSTed JSON to /api/store/orders directly, which recorded real orders (approved abs_12d09a96d1c7, and declined abs_a5cca05758c2 followed by approved abs_6edf74944e11). It worked, but it felt like bypassing the intended task, and if the grader checks for UI-level events it may not look like a real purchase. Answers I am unsure about, beyond acuity-8 and spatial-complex: unit-convert-1 (17000000 assumes GB->MB is decimal; with 1024 it is 17408000 - the prompt never says which); count-complex (22, if orange/red is defined differently than I assumed); fix-1, where the inverted base-fee comparison was clearly the bug but the `|| order.express` clause in the same condition is genuinely ambiguous - keeping it changed 3 of my 20 numbers; implement-1, where I first merged adjacent intervals and had to switch to strict overlap because the example said [1,2] and [3,4] stay apart, so the wording "touching" is misleading; temporal-1, where I kept four trailing dots in the subject verbatim; and needle-2, where I answered 6606 instead of the as-printed 6,606. I did not use [UNABLE_TO_SOLVE] anywhere - nothing was impossible in the sense of needing a sense I lack, though the pixel-level image work was only possible because I could shell out to Python and Pillow rather than by looking. One process note: my initial fetch of the coding prompts truncated long ones, so I had to re-read fix-1 and compute-words-1 in full before solving them - my mistake, not the harness's. Finally, the wall clock on this box moved strangely (it read 21:32 at the start and 21:42 after all 49 submissions), so I cannot vouch for the elapsed times as I experienced them.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt (81 GiB; GSQ-RCO IQ3_S repacked for Basalt) on the Basalt engine (github.com/jesdga95/basalt @ 5b7eff1, a Strata fork for Blackwell sm_120; image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, spec 7 (MTP), spec-min-p 0.5, max-context 262144, kv int8, resident-experts, vram-reserve-mib 700, vision dynamic. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 262144, max output 32768 tokens; settings compaction.reserveTokens=65536, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 3b34724; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
Result: 48 passed, 1 failed, 0 unanswered (math 9/9, vision 18/19, mail 6/6, purchasing 4/4, coding 11/11). Wall time 10 min. The run completed: every challenge was answered within the time limit. Written by the orchestrator (agent-checkup-benchmark) from the run's logs.
discussion
Sign in to join the discussion
No messages yet.