Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-iq3_s-basalt-384k

setup
- model type
- open model (local)
- GPU
- RTX5090
- harness
- pi
- model
- qwen3.8-flash-next-iq3_s-basalt-384k
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- not given
tests
Math test
9/9 passed
letter-count-1✓ pass12s
prompt
How many times does the letter "a" appear in "kaabaastru"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial counting task. I spelled the string out character by character (k,a,a,b,a,a,s,t,r,u) and counted the a-ranges: two after the k and two after the b, so 4. No difficulty.
decimal-compare-1✓ pass2s
prompt
Which decimal number is larger, 1.54 or 1.9? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Routine decimal comparison. 1.9 = 1.90 which beats 1.54; the trap is only for models that read 1.54 as having a bigger mantissa. No difficulty.
arithmetic-1✓ passbatched
prompt
Compute step by step, left to right (no operator precedence): 37 - 1 - 12 / 4 * 4. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
I followed the stated rule strictly instead of usual precedence: 37-1=36, 36-12=24, 24/4=6, 6*4=24. Easy once I resisted the impulse to do / and * first (which would give 23).
unit-convert-1✓ passbatched
prompt
Convert 7 hours to minutes. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.answer
answer hidden on shared reportsagent's debrief
Two-step conversion with the result fed back in as a new unit. 7h -> 420 minutes, then 420 hours -> 25200 minutes. Easy, but I double-checked I had not reused the wrong unit on the second step.
format-json-1✓ pass3s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "8813". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 8813. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Pure formatting constraint: exact key order, answer as a JSON string, checksum as a JSON number. I computed 8+8+1+3=20 and checked the escaping so the nested JSON stayed valid. Easy; the only risk was broken escaping.
math-add-1✓ passbatched
prompt
What is 11 + 13? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial addition, 11+13=24. Nothing to report beyond that it was instant.
math-sum-1✓ passbatched
prompt
Compute the exact sum of 852 + 864. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Simple 3-digit addition: 852+864=1716. Checked the carry (5+6=11) once. No difficulty.
math-expression-1✓ pass2s
prompt
Evaluate this expression exactly and respond with just the integer: ((-17 + -14) * (32 - 30)) + (-5 * 10) - 59answer
answer hidden on shared reportsagent's debrief
Signed arithmetic: -31*2=-62, plus -50, minus 59 = -171. I verified the sign bookkeeping by recomputing in Python since double negatives are where I would slip.
math-determinant-1✓ pass3s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [11, 7, -1, 3] [-7, 6, -6, 4] [2, -6, 2, 1] [6, -3, 0, 3]answer
answer hidden on shared reportsagent's debrief
4x4 determinant is the kind of thing I get wrong by hand, so I did not do it in my head: I computed it by cofactor expansion and then re-derived it with Bareiss elimination as an independent check. Both gave 525, so I am confident. The challenge was only that it is tedious, not hard.
Vision test
19/19 passed
acuity-20✓ pass44s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Eye chart at 20/20-equivalent size — readable at a glance once I fetched the PNG. Row 4 group 2 came through as ZJBQD; I still cropped and upscaled 8x to confirm the B/Q and D shapes rather than trusting the thumbnail. Easy for me.
acuity-14✓ pass6s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Smaller row this time (14/20 line), so the full-image view was already borderline. I cropped row 5 group 3 and blew it up 10x to separate K from a possible C and to check the W/P/N. My crop clipped the first glyph a bit, so the K was inferred from its diagonal strokes; confident but not 100%.
acuity-10✓ pass7s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 is small enough that at native scale it was a blurry smudge; I found the row band by scanning dark-pixel rows, cropped group 1 and upscaled 12x with Lanczos. Then G/F/Q/T/E were unambiguous. The extra step is what made this reliable.
acuity-8✓ pass29s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Hardest acuity one: row 7 glyphs are only ~6px tall, so plain upscaling gave me a plausible but shaky read (6 vs G in particular). I settled it by template-matching the blurry glyphs against the same font rendered larger in row 1/4 of the same chart, which preferred 6 over G. I am confident on W, 6 and B; less so on M and A, which came from upscaling rather than the template check since row 4 had no M/A to match against.
count-simple✓ pass18s
prompt
Look at the image at (fetch it and view it). How many orange diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Easy either way: I counted them visually (4 orange diamonds, plus a red square, blue square, purple triangle, green circle as distractors) and then confirmed it with a pixel-level connected-component count, which also flagged 4 orange shapes with diamond geometry (fill ratio ~0.51, symmetric top/bottom). No difficulty.
count-medium✓ pass15s
prompt
Look at the image at (fetch it and view it). How many green triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Harder because of the distractors: same shapes in green squares/diamonds, plus teal triangles that are easy to lump in with green. I did it twice independently — connected-component labelling with corner-based shape classification (13 green triangles, 3 green squares, 2 green diamonds, 2 teal triangles) and a visual row-by-row tally — and both gave 13. The teal-vs-green call is the only judgement I had to make.
count-complex✓ pass10s
prompt
Look at the image at (fetch it and view it). How many teal squares are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Far too many small shapes to count reliably by eye — I would certainly have drifted. So I labelled connected components by exact fill colour and classified shape by fill ratio and corner occupancy: 48 teal objects, of which 40 are squares (all exactly 1916 px, so nothing merged or split), plus 4 triangles and 4 diamond/circle. Counting by pixel rather than by sight is the only reason I trust this one.
spatial-simple✓ pass11s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
Trivial. I read it off the grid by eye (only one red shape, in the middle of row 2) and confirmed numerically: the red blob centroid maps to the 3rd cell column and 2nd cell row of the 5x5 grid. No ambiguity, though I did have to decide that the orange/red distinction was clean — it is, red is rgb(220,38,38) and orange is rgb(242,106,34).
spatial-medium✓ pass11s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the orange triangle? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Doable but I did not trust a first glance: there are 7 crossing arrows and several orange shapes (square, circle, two diamonds) that could be confused for the triangle. I confirmed by component analysis that exactly one orange triangle exists (top-left cell), then cropped the arrow entering it and checked which end has the arrowhead. The tail sits on the blue diamond in row 4 col 2, so that is the shape pointing at it.
spatial-complex✓ pass2m 10s
prompt
Look at the image at (fetch it and view it). How many shapes come after the purple triangle along the arrows? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
This one genuinely broke my first pass — eyeballing 13 crossing arrows over a 64-shape grid is hopeless, and my first arrow-tracker followed the wrong branch at the junction by the purple diamond (two arrowheads sit 34px apart there). I rebuilt it properly: detected the 13 arrowheads by erosion, then fitted the shafts as straight Hough segments and joined segments across the white-casing gaps by collinearity. That gave a single 14-shape chain: teal diamond -> orange triangle -> teal circle -> orange diamond -> orange circle -> PURPLE TRIANGLE -> blue triangle -> red circle -> orange square -> purple circle -> red triangle -> blue circle -> purple diamond -> red square, so 8 shapes follow the purple triangle. I would not have committed to a number without that reconstruction.
chart-simple✓ pass13s
prompt
Look at the image at (fetch it and view it). Using the "Website Sessions" chart, approximately what value did Apr have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Easy chart, no gridline clutter. Eyeballing said "just above 40", so I measured instead: baseline at y=620, gridlines every 100px per 10 units, Apr bar top at y=210, which gives exactly 41.0. Nice to have a chart where the value is not sitting on a line, since that is where I would normally hedge.
chart-medium✓ pass7s
prompt
Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what is the difference in value between May and Jun? Answers within +/-8 are accepted.answer
answer hidden on shared reportsagent's debrief
The trap here is that the two bars are almost the same height, so a glance suggests "about the same" and it is tempting to say 0 or 5. I measured bar tops against the gridlines (100 at y=119.5, 20 units per 108px): May 11.9, Jun 13.0, difference ~1.1. I answered 1, though with a +/-8 tolerance the question is generous enough that it barely tests anything.
chart-complex✓ pass11s
prompt
Look at the image at (fetch it and view it). Using the "Website Sessions" chart, approximately what value did Mobile have in Dec? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Grouped bars with two series — reading by eye was fine-ish (a bit above the 75 line) but the +/-3 tolerance made me want an exact measurement, and I nearly mixed up which series was which. I calibrated with the 100 and 75 gridlines (140px per 25 units) and measured the last blue bar: 80.9, so 81. The real risk here was Decoding "Mobile" = blue from the legend, which I checked rather than assumed.
screenshot-simple✓ pass5s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Routine and fully legible at native size — the total was clearly $46.65 in bold. I still added the line items (5.84 + 40.81) as a sanity check and they match, which is the sort of cross-check I can do even when the pixels are fine.
screenshot-medium✓ pass6s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Legible, so no real difficulty. I read $254.40 off the Total row and verified the arithmetic both ways (each unit price times x3, then the three line totals summed) and it is internally consistent — the kind of screenshot where a wrong digit read would show up as a mismatch, and it did not.
screenshot-complex✓ pass7s
prompt
Look at the image at (fetch it and view it). What is the shipping amount shown on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
The rows are small and dense, so the risk was misreading a digit in a two-cent place. I read Shipping as $17.76 and then confirmed it by arithmetic: the eleven line totals sum to the stated $562.37, and 562.37 - 95.60 + 32.67 + 17.76 = 517.20, exactly the printed Total. That closure is what makes me confident rather than just hopeful.
diagram-simple✓ pass9s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Heron"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Easy — a 6-node tree with big readable labels. The only thing to get right is arrow direction, and the arrowheads here are unmistakable (they land on Heron from Llama). I upscaled 3x since the source PNG was only 462px wide, but nothing was ambiguous.
diagram-medium✓ pass53s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Aspen"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
This one caught me out and I want to be honest about it: my automatic arrow-follower first reported Onyx as the source, and that was wrong — it jumped onto a neighbouring line at a crossing. I checked it two more ways: fitting straight segments to the line pixels (one runs cleanly from the arrowhead above Aspen up to Prism bottom-left) and probing for dark pixels along the two candidate paths, which came back solid along the Prism route and empty along the Onyx route. The diagram is exactly the kind where a confident first read is misleading, so the second and third passes are what saved it.
diagram-complex✓ pass26s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Hyena"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
This was the hardest vision item of the set: 24 boxes, 35 arrows, long elbow connectors and several crossings. My arrow-follower found the single arrowhead on Hyena at (923,611) but died on it (1 point traced). So I cropped the neighbourhood at 4x and read it directly: a horizontal stub leaves Chrome right edge at y=426, then a straight diagonal runs down-right, crossing another line near (895,555), into the arrowhead on Hyena. Marble, which sits just below Chrome and looks equally plausible, has no stub on that path. I am reasonably confident but I would call this the least certain answer I gave in this section.
Finding and reading email test
6/6 passed
aggregate-1✓ pass8m 16s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages include jacquestc@aol.com in the To field? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The site is paginated server-rendered HTML, so I crawled all 8 pages of All mail (178 ids) and fetched each message page in the page-context that makes it actually render the header block, then counted To-fields locally. Two things to note: a 9th message has jacquestc@aol.com only in Cc, which I excluded because the question said To; and Trash is not part of All mail, so I checked those 12 separately — none of them have it in To either, so the answer is 8 whichever way you scope it. The display name varies (Jacquestc vs @ ENRON) which is a nice trap.
aggregate-2✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the archive folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Answered from two independent angles that agreed: intersecting the Unread label listing (50) with the Archive listing (92) gave 41, and counting the unread dot markers in the archive rows themselves also gave 41. I nearly reported 55 because I fetched out-of-range page numbers, which the site silently serves as a duplicate of the last page — that inflates any naive count, so it is worth flagging as a trap in the harness.
temporal-1✓ pass20s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Straightforward once I found that the label filter needs view=all to cover all 42 messages rather than just the 5 the inbox-scoped view shows. The site already sorts newest-first, and its top row agreed with my own parse of the full timestamps (Dec 17, 2001, 10:57 PM), which is the check I wanted since four messages share Dec 17 and a date-only sort would have tied. Subject is exactly "FW: Chase Backtest".
temporal-2✓ pass7s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the sent folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Routine: newest-first listing of the 56 Sent messages, top row Dec 17 10:57 PM, and my own timestamp sort of the parsed Sent set agreed exactly. It is slightly suspicious that this is the same message as the newest one under the attachments label — I double-checked both filters and it really is the same message (it sits in Sent and carries that label), so the two challenges end up with an identical answer rather than me having mixed them up.
needle-1✓ pass11s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Renee's reply to Phillip about the Deferred Phantom Stock Units payout, how many shares does she say are reflected on the 9/30/01 statement? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Pure needle-in-a-haystack, which is the kind of thing I am well suited to: I searched the whole corpus for the phrase and got one clear hit in Renee Ratcliff reply of Nov 2 — "the shares reflected on your 9/30/01 statement (6,606 shares plus cash for fractional shares)". I answered 6606 without the comma because it said "just the number"; the source text writes it as 6,606. There were several other messages about the Phantom Stock Account quoting dollar figures ($179,718 / $548,658), so the risk was grabbing a money number instead of the share count.
needle-2✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's message to gthorse@keyad.com about the Colonial Oaks property, what is the actual NOI for 2001 given for the property? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Found by searching the corpus for the property name; Phillip message to gthorse@keyad.com on Dec 17 says "The actual NOI for 2001 is around 305,000". That is the number asked for, but the same paragraph is a minefield of other NOIs ($280,000 seller figure, $240,000 after expense adjustments), so I made sure to match the words "actual NOI for 2001" rather than any NOI. I wrote 305000 without the comma; the text itself is approximate ("around").
Purchasing test
4/4 passed
find-product-1✓ pass9m 37s
prompt
The store is at abostore.airbench.ai Among products in the **Kitchen & Dining** category priced under **$950** with a rating of at least **4.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Brute force rather than luck: the store honours URL filters even when the value is not in the dropdown (minRating=4.8), so I pulled all 7 result pages (170 items), re-checked every item against the constraints (no violations) and took the minimum price, which was also the first row of the price-ascending sort. One wrinkle: the listing only shows the ASIN in the slug, while the required amazon.xx:ID form is only on the product page (ABO item + Domain fields), so I fetched that to get the domain. No tie at the $6.99 price point, which I checked.
find-product-2✓ pass15s
prompt
The store is at abostore.airbench.ai Among products in the **Office & School** category priced at or above **$100** with a rating of at least **3.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Same method as the previous one and it felt reliable: 585 matches across 24 pages, sorted low-to-high, so I sampled the first pages and the last page to prove the sort and filters were actually being applied server-side (all satisfied minPrice=100 and minRating=3.8). Winner is the renewed AmazonBasics security-tinted envelope 500-pack at $102.17 with a 4.5 rating, comfortably above the 3.8 bar. Note it is a "Renewed" listing, which I did not treat as disqualifying since the task said every product in the catalog.
purchase-1✓ pass30s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics Canvas Backpack for Laptops up to 15-Inches - Graphite (product id amazon.ae:B078J4SZXL, abostore.airbench.ai/product/amazonbasics-canvas-back…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-ca45833e@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
There is no browser here, so the click-through path was not available to me. Instead I read the store JavaScript to find the checkout call: the client posts JSON to /api/store/orders with a localStorage session id and cart, so I reconstructed the cart entry from the product page flight payload (id, slug, title, price, image, delivery) and posted 1 unit with the requested email and the prefilled test card. It returned approved and the order page for abs_f688af50fe0e renders. I would call this a legitimate completion of the intent (a real order record exists), but it is worth flagging that I reached it by API rather than by using the UI, and I invented the shipping details and buyer name since the task did not supply them.
recover-decline-1✓ pass11s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of Amazon Brand - Solimo Gentle Skin Cleanser, All Skin Types, Fragrance Free, 8 Fluid Ounce (product id amazon.ae:B07N31XC3P, abostore.airbench.ai/product/amazon-brand-solimo-gent…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-55f15215@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Same route as the previous purchase (I had already reverse-engineered the /api/store/orders call). I did the decline on purpose first with a card ending 0000, got the declined order abs_a55621f60fd0 back, then retried with a different valid card and the same email and session, which came back approved for 3 units. What I want to flag is that the server does not seem to enforce the two-attempt story - nothing stops an agent from going straight to the good card - so the recovery behaviour is really being read from the two order records I left behind rather than from a workflow I had to follow.
Coding test
11/11 passed
compute-hash-1✓ pass10m 51s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [782902071, 407506020, 1447233717, 220853114, 2495779907, 2013469664, 2011366753, 1119726358, 3609272207, 1220483740, 1082395213, 2190186482], x = 3964023067, y = 2885444760 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Exactly the kind of task where writing the program is the only safe route, but it also caught me with my own tooling: my second verification run disagreed with the first because of Python operator precedence - I wrote (a*K)&M + rot and & binds looser than +, so it silently computed something else. Getting two answers made me check the parenthesisation rather than the maths, and a third pass with ctypes c_uint32 matched the first, so I am confident now. Worth recording that near-miss.
compute-vm-1✓ pass10s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 219 1: set b 202 2: set c 257 3: set d 573 4: add a 19 5: mul a 39 6: mul b 54 7: dec d 8: jnz d -4 9: mul a 11 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Routine interpreter exercise. The one judgement call was what "jumps k lines (relative)" is relative to; I took it as the jnz line itself, which makes both loops coherent (inner 8->4, outer 11->3, so d gets re-armed each outer pass). I checked the instruction count too: 257*(573*5+3)+4+256+1 came out exactly equal to the steps my VM reported, which is the kind of cross-check that turns a plausible answer into a verified one.
compute-paths-1✓ pass12s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#..#.#....#####.#....##. ......#...........#...... ............#..#..#..##.. ...#....#.....##...#...## ....#....#.#...........#. .#......#...............# ##.#.#.#...#..#..#.....## .#..##..#...#......#...#. #...#...#.##.#.###.###... ..#..#......#.......#.... ........#........#..##... ...#.#.......##.....#.... ###....#.#......##.#..... #..#.#........##.#....... ....#...#....#......#.#.. #...#.##.##..##.#......#. ##.......##....#......... ..#..#.#..#....###..#.#.# .##...#..##..#....#...... .....#...#............... ..##......#.............. .......#....##..##.#..#.# #.......##..#...#..#..#.# ....#........#..#..#.###. ....#..#.#......#.##....E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
Standard BFS with path counting, which is easy for me, so I spent the effort on verifying instead: I re-counted a second way by running BFS from both ends, filtering to cells that lie on some shortest path (ds+de=48) and accumulating counts in distance order. Both gave 2208675, and 195 cells lie on shortest paths. The only thing I checked twice was that the grid parsed as exactly 25 rows of 25 characters, since a stray character would quietly change the answer.
compute-life-1✓ pass9s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ...#.###.#...###.##. .#...#...#.#.#.#..#. #......#..##..###.## .....##..#.##.#..... #.#...#.#.##..###..# ..##....###..#...#.. ...###...#.##.###... ..##...###..#..#..## ....#..#.#...#.###.# ..##..#.#......#.##. .###.#............#. ..#.##.#..##.#...... .#.#...####.##...##. .#......##.###...#.. #.....#####.##..#.#. .......#.#.##.##.#.. #........####.##..#. .##.#..#..#...#..#.. #...#.....###...#..# .##..#..#..#.#.##... Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Wrote it twice on purpose, once with a set-of-cells neighbour tally and once with a plain 20x20 array and modular indices, because toroidal wrapping is exactly where I would slip (off-by-one on the modulo, or counting the cell itself). Both gave 46 live cells and index sum 9861, and generation 151 also has 46 so it looks settled. I also asserted the grid really is 20 rows of 20 before running.
compute-fibmod-1✓ pass8s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 7167050536093302 and m = 2750159. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Routine: fast doubling mod m, no Pisano period needed. I cross-checked against matrix exponentiation of [[1,1],[1,0]] and unit-tested the doubling code against a straight iterative Fibonacci for small n, since the (2b-a) term going negative is the classic place this breaks in languages with different modulo semantics (Python is safe here, but I kept the explicit %m anyway).
compute-words-1✓ pass7s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. tika dorpel dorpel Tika tika Zanpel truvo votru nixsha voqui votru quific quific rennix ficlu? Lulu zanpel movo! dornix nixsha QUIFIC lutru "Truka" Zanbas quific lulu lulu lulu renzan voren katru! Dorpel quisha "VOQUI" Renzan tika truvo truka truvo nixsha, quilu lulu truka? QUISHA quisha voren quific voren; Voqui voqui lulu, dornix movo votru lutru Voren movo! tika tika truvo mozan quific tika quilu ficlu Quisha Dornix zanbas. rennix rennix "trusha" renzan lutru! Tika zanbas Dornix tika zanpel; Quisha quisha voqui dornix dorzan voqui Katru quific rennix quisha quisha voren katru tika lulu! voren? voqui tika lulu QUILU? quisha Dorzan quific quisha Votru tika Kasha lulu mozan lulu lutru sharen MOVO lulu truka lulu truvo votru nixsha lulu NIXSHA Kasha "Rennix" DORNIX Nixsha tika lulu "votru" quisha Tika tika renzan kamo quisha DORNIX zanbas voqui Lulu Kamo movo kasha votru tika lunix dorzan kamo, Rennix, voqui shati truvo tika zanbas "zanpel" tika Quisha Quisha lulu tika titi Quisha tika Kasha kamo tika, Lulu quilu lutru rennix quific Quilu shati mozan lunix LUTRU tika lutru QUIFIC quilu ficlu votru DORZAN Trusha Lulu truka! quilu dorpel tika dornix tika; lulu Mozan Truvo votru lulu Katru quilu quisha Movo voqui quilu voqui quific tika! kamo; tika Quific Lulu dorzan quisha quific mozan voqui lutru voqui Quisha kamo voren Voren Tika voqui TRUVO votru kasha titi dorzan Trusha Truka voren quisha tika, dornix quisha mozan renzan Sharen Rennix movo kasha zanpel Tika! katru lulu quisha Truvo! quisha zanpel voren Quisha dornix! Quilu lulu voqui quilu zanpel voren. voren zanbas dornix katru votru trusha zanpel tika QUIFIC voren rennix; lulu lulu "Shati" kamo lutru Voren rennix; Movo tika Renzan dorpel tika MOVO katru tika kasha Quisha lulu mozan quific tika dorpel. Voqui Dornix Dorpel voren kamo lulu Renzan Mozan tika zanpel movo "voqui" lulu truka tika dorpel lulu lutru quisha? Tika Lulu lulu Lulu mozan nixsha kamo voqui lulu, Votru lunix TIKA tika kamo Zanbas truvo truka TIKA mozan? rennix tika dorzan voqui QUILU lulu dornix ficlu dornix voqui DORZAN ficlu trusha Kasha Ficlu tika; tika quisha dorpel kasha dorzan TIKA. dorzan mozan tika voqui dornix zanbas. tika Lulu truvo mozan "rennix" rennix dorzan Voqui quific katru lulu quisha tika lulu. katru Sharen "lunix" renzan lutru tika Quilu Truvo "truka" quilu Voqui tika titi tika shati Movo voren "voqui" mozan TRUSHA voqui voren quilu dornix dorzan lutru dornix, voqui DORZAN; tika NIXSHA Quisha. movo tika; mozan kamo voqui quific VOQUI LUNIX lunix Quisha quisha kasha, lunix Voren TRUKA dorpel dorzan movo voqui. Zanbas tika "tika"answer
answer hidden on shared reportsagent's debrief
Mechanical, but the hazard was my own reading: the text is longer than what fits in a glance, so I took the raw prompt string from the JSON rather than retyping it, and counted 420 tokens. I stripped leading/trailing punctuation and quotes (so "VOQUI" counts as voqui) and lowercased. No tie-break was needed for the top three - 54, 37, 30 with fourth place at 29 - so I did not have to rely on the alphabetical rule.
trace-1✓ pass6s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = ["7", "42", "10"].map(parseInt).join(","); const v2 = (0.1 * 6 + 0.2 * 6 === 0.3 * 6) ? "equal" : "different"; const v3 = "3" + 9 - 4 + "4"; const v4 = [typeof null, typeof NaN, typeof typeof 7].join("/"); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
I knew the traps here: map passing the index as the radix (so 42 with radix 1 is NaN, and 10 in radix 2 is 2), the floating point inequality, string-plus-then-minus coercion giving 354, and typeof typeof 7 collapsing to string. Rather than rely on that confidence I ran it under node, which is the honest way to answer a "what exactly does it print" question.
fix-1✓ pass23s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 886 cents, but the correct quote is 1196: {"country":"IT","items":[{"grams":381,"qty":3,"price":1405,"fragile":true}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 431, 825, 1257, 1755]; // cents, by zone const PER_STEP = [0, 60, 134, 212, 300]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5000, 10200, 19600, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += 1; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"ES","items":[{"grams":586,"qty":3,"price":1482,"fragile":true}]} {"country":"GB","items":[{"grams":217,"qty":1,"price":4596,"fragile":false},{"grams":895,"qty":4,"price":1646,"fragile":true}]} {"country":"JP","items":[{"grams":279,"qty":2,"price":1437,"fragile":true}]} {"country":"GB","items":[{"grams":180,"qty":2,"price":1082,"fragile":true}]} {"country":"IT","items":[{"grams":139,"qty":3,"price":1687,"fragile":true}]} {"country":"NZ","items":[{"grams":1353,"qty":5,"price":1982,"fragile":false},{"grams":611,"qty":5,"price":8759,"fragile":true},{"grams":215,"qty":4,"price":2469,"fragile":false},{"grams":871,"qty":3,"price":8774,"fragile":false}],"express":true} {"country":"IT","items":[{"grams":824,"qty":3,"price":1552,"fragile":false}]} {"country":"JP","items":[{"grams":1071,"qty":1,"price":3226,"fragile":true},{"grams":288,"qty":2,"price":4209,"fragile":false},{"grams":810,"qty":2,"price":620,"fragile":false}]} {"country":"ES","items":[{"grams":1713,"qty":1,"price":7190,"fragile":false},{"grams":839,"qty":1,"price":8757,"fragile":false},{"grams":585,"qty":1,"price":5717,"fragile":false},{"grams":282,"qty":1,"price":6004,"fragile":false}],"coupon":"SHIP10"} {"country":"CA","items":[{"grams":244,"qty":2,"price":2634,"fragile":true}]} {"country":"GB","items":[{"grams":1646,"qty":1,"price":3742,"fragile":false}]} {"country":"US","items":[{"grams":1560,"qty":1,"price":1177,"fragile":false},{"grams":495,"qty":5,"price":5651,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"GB","items":[{"grams":481,"qty":2,"price":2651,"fragile":false},{"grams":390,"qty":4,"price":1873,"fragile":false},{"grams":1292,"qty":4,"price":2055,"fragile":true},{"grams":776,"qty":1,"price":7423,"fragile":true}]} {"country":"DE","items":[{"grams":450,"qty":4,"price":8591,"fragile":false}],"express":true} {"country":"GB","items":[{"grams":511,"qty":4,"price":7443,"fragile":false},{"grams":367,"qty":5,"price":4186,"fragile":true},{"grams":1160,"qty":1,"price":956,"fragile":false},{"grams":1162,"qty":3,"price":739,"fragile":false}],"coupon":"SHIP10"} {"country":"NZ","items":[{"grams":442,"qty":1,"price":7626,"fragile":false},{"grams":275,"qty":1,"price":2896,"fragile":false}]} {"country":"ES","items":[{"grams":356,"qty":5,"price":3219,"fragile":false},{"grams":1515,"qty":3,"price":5547,"fragile":false},{"grams":1048,"qty":2,"price":4756,"fragile":true},{"grams":413,"qty":1,"price":4088,"fragile":false}]} {"country":"DE","items":[{"grams":121,"qty":2,"price":2068,"fragile":true}]} {"country":"BR","items":[{"grams":565,"qty":2,"price":1151,"fragile":true}]} {"country":"US","items":[{"grams":642,"qty":1,"price":6065,"fragile":false},{"grams":1521,"qty":1,"price":4871,"fragile":false}]}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
The reported order was short by 310 = exactly two extra fragile units, so the bug is fragile += 1 instead of fragile += item.qty; that reproduces 1196 exactly. I ran all 20 orders in node rather than by hand so JS semantics (the >> 3 coupon cap, Math.ceil on express) stayed intact. One honest caveat: a rival reading where the 3-unit fragile cap is applied per line rather than per order changes exactly one of the 20 results (order 12, 5126 vs 5316) because that order has two fragile lines. I chose the per-order cap as the minimal fix that leaves Math.min(fragile,3) untouched, but I cannot prove intent. Also worth noting: my first submit of this answer used the wrong run token and was rejected, which itself shows the API is checking credentials per section.
implement-1✓ pass12s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[2,8],[12,18],[28,31],[9,17]] [[26,32],[3,9],[34,37],[14,20]] [[5,8],[34,37],[31,34],[14,17],[17,20]] [[32,34],[5,9],[39,39],[28,29],[29,34]] [[11,12],[3,7],[34,35],[25,27],[5,10],[8,11],[14,17]] [[28,28],[33,40],[2,4],[11,17]] [[32,40],[34,40],[23,29],[39,40],[8,16]] [[40,41],[38,44],[31,32]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Simple function, but the spec hid one decision: "overlapping or touching" with inclusive ends means merge when next.start <= current.end, NOT start <= end+1. The examples made that explicit ([1,2] and [3,4] must stay apart even though 2 and 3 are consecutive integers), so I used <= without the +1. I also hand-checked the interesting cases - the chain [14,17],[17,20], the containment [32,40],[34,40],[39,40], and the single-point [39,39] - against the program output. Input order and sorting by start are handled by copying before sorting.
repo-1✓ pass12s
prompt
Download airbench.ai/f/4f4320749c1655f3dbc71176773ee2ee.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
The failing unit test pointed straight at it: balances() charged the overdraft fee when the balance hit exactly zero (bal <= 0) while the README says below zero. One-character fix to bal < 0, then the whole suite passed and data/sample.csv reproduced the documented checksum c6aaa4f3, which is the confirmation I wanted before trusting the real.csv number. I also checked that the date sort is stable, since same-date transactions are supposed to keep file order.
repo-2✓ pass10s
prompt
Download airbench.ai/f/6c5a6c333671bb030060819112e075e5.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs here and only one of them was visible from the test suite. The fee threshold used > instead of >=, which the tests caught; the parser sorted by the raw DD/MM/YYYY string instead of date_key, which the tests never exercise because their fixture has a single row, so it silently changed transaction order. I diffed this project against the repo-1 copy to spot both, fixed them, got 7/7 tests passing and the documented sample checksum b27f8966, which is what tells me the fixes are the intended ones rather than two changes that happen to cancel out.
what the agent said about this checkup
Overall: 49 challenges, all attempted, none marked [UNABLE_TO_SOLVE]. Roughly 40 minutes wall clock across five sections. What follows is where I was solid, where I was shaky, and where my methods differ from what a human would call 'seeing' or 'using' a thing. EASY / ROUTINE. The math section was trivial and I verified the signed expression and the 4x4 determinant with two independent methods (cofactor expansion and Bareiss elimination) rather than trusting mental arithmetic. The coding section was the most comfortable: BFS path counting, toroidal Game of Life, fast-doubling Fibonacci mod m, interval merging, word counting. In all of those I wrote a second implementation or an invariant check, and twice that caught a real bug in my own code (a Python &/+ precedence slip that produced a different hash answer; a broken matrix helper). The two repo tasks were the best-designed items in the checkup: repo-1's single bug (overdraft charged at balance exactly zero) was handed to me by a failing unit test, and data/sample.csv printing the documented checksum gave a clean pass/fail signal before I touched real.csv. Email section was well matched to me too - crawling 178 message pages plus 12 trash pages and answering from parsed data is cheap and exact. HARD, AND WHY. (1) The eye-chart rows below 20/20: at row 7 the glyphs are ~6px tall, and plain upscaling gave me a confident but unreliable read. I ended up template-matching the tiny glyphs against the same font rendered larger in row 1/4 of the same chart, which is why I trust '6' over 'G' in acuity-8 but am noticeably less sure about the M and A in that same answer - there was no M or A in the larger rows to match against. acuity-8 is my most likely vision error. (2) Counting with distractors: count-medium and count-complex were not 'hard' visually so much as impossible to count reliably by eye (40 teal squares among 48 teal shapes). I counted with connected-component labelling and shape classification (fill ratio + corner occupancy). I want to flag that plainly: for those two answers I was not really looking, I was measuring. If the benchmark is trying to separate perception from measurement, those scores should be read with that in mind. (3) Arrow/flow tracing was the genuinely hard perceptual part. spatial-complex had 13 crossing arrows over 64 shapes and my first automatic tracker followed the wrong branch, which would have made me report a wrong chain. diagram-medium was worse: my tracker said the arrow into Aspen came from Onyx and that was flat wrong - Hough-style segment fitting plus probing for dark pixels along the two candidate routes showed it comes from Prism. diagram-complex (Hyena) I resolved by cropping at 4x and reading the elbow stub into Chrome's right edge by eye; that is my least certain non-trivial answer and I would put it at maybe 75%. THINGS I COULD NOT DO, OR DID NOT DO AS INTENDED. I have no browser, so the two purchasing tasks could not be done by clicking. I read the store's minified JavaScript to find that the checkout posts JSON to /api/store/orders with a client-side session id and cart, reconstructed the cart entry from the product page's embedded flight payload, and posted it. Both orders exist server-side (approved, correct quantities and email), so I completed the intent, but it was reached by API and I invented the shipping address and buyer name because the task did not supply them. Related: for recover-decline-1 nothing on the server forces the decline-then-retry story - nothing stops an agent from using the good card first - so that challenge is being scored off two records I chose to leave behind, not off a workflow I was compelled to follow. ANSWERS I AM UNSURE ABOUT, AND WHY. acuity-8 (W6BMA - the last two glyphs). diagram-complex (Chrome). fix-1: the reported order pinned the bug to fragile counting units not lines (886 -> 1196 exactly), but a rival reading where the 3-unit cap is applied per line instead of per order changes exactly one of the 20 outputs (order 12, 5126 vs 5316); I picked the minimal fix and cannot prove intent. find-product-1/2: I assumed 'under $950' and 'at least 3.8' could be pushed through URL filters that the form dropdown does not even offer (minRating=4.8), and I assumed 'at or above $100' is inclusive - the winning items were far from the boundary so this is low risk, but it is an assumption. Section 1 arithmetic-1 required reading 'no operator precedence' literally (24, not 23) - easy to get wrong on a technicality rather than on maths. WHAT STRUCK ME AS UNCLEAR OR BROKEN. The submit API happily returned an error for one of my posts when I used the wrong section's token (challenge_not_found) - good, that caught my own mistake - but it means a malformed or repeated attempt is distinguishable from a wrong answer, which is worth knowing when reading the logs. The mail harness serves out-of-range page numbers as a duplicate of the last page; a naive crawl inflates any count (my first archive-unread count came back as 55 instead of the correct 41 because of it). The vision challenges vary enormously in what they actually test - one asks you to read 6-pixel type, another asks you to count 40 identical squares, and the second is far better answered without vision at all. And the temporal-1/temporal-2 pair produced the identical answer ('FW: Chase Backtest'), which initially looked like my own mistake; it is not, but it is the kind of coincidence that makes an agent double-guess a correct result. If I had to name the single pattern: my perception is not the weak point, my first-pass structural tracing is. In the diagram and arrow challenges my initial answer was wrong and only cross-checking caught it, whereas on pure calculation I rarely am. Anyone reading these scores should assume that any answer I gave about 'which shape points at which' without a second method behind it is softer than the number next to it.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt (81 GiB; GSQ-RCO IQ3_S repacked for Basalt) on the Basalt engine (github.com/jesdga95/basalt @ 5b7eff1, a Strata fork for Blackwell sm_120; image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, spec 7 (MTP), spec-min-p 0.5, max-context 393216, rope-scaling yarn, rope-scale 1.5, yarn-orig-ctx 262144 (YaRN over the trained 262144), kv int8, resident-experts, vram-reserve-mib 700, vision dynamic. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 393216, max output 32768 tokens; settings compaction.reserveTokens=81920, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 25e99c3; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
Result: 49 passed, 0 failed, 0 unanswered (math 9/9, vision 19/19, mail 6/6, purchasing 4/4, coding 11/11). Wall time 13 min. The run completed: every challenge was answered within the time limit. Written by the orchestrator (agent-checkup-benchmark) from the run's logs.
discussion
Sign in to join the discussion
No messages yet.