Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-iq3_s-basalt-384k

setup
- model type
- open model (local)
- GPU
- RTX5090
- harness
- pi
- model
- qwen3.8-flash-next-iq3_s-basalt-384k
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- not given
tests
Math test
9/9 passed
letter-count-1✓ pass12s
prompt
How many times does the letter "l" appear in "molpelllllu"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial character-counting task. I did not eyeball it because I have a reputation with off-by-one errors on runs of identical letters; I extracted the exact string from the API JSON and counted with code to be safe.
decimal-compare-1✓ pass3s
prompt
Which decimal number is larger, 4.3 or 4.86? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Easy: 4.86 > 4.3 once you line up the place values (4.30 vs 4.86). The only trap here is treating the longer mantissa as automatically bigger, which happens to give the right answer in this direction.
arithmetic-1✓ pass3s
prompt
Compute step by step, left to right (no operator precedence): 36 + 16 - 18 * 2 - 14. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
The instruction to ignore operator precedence is the whole challenge — my instinct is to do 18*2 first, which would give 22 instead. I stepped it strictly left to right: 52, 34, 68, 54.
unit-convert-1✓ pass2s
prompt
Convert 12 kg to g. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.answer
answer hidden on shared reportsagent's debrief
Routine two-step chained conversion: 12 kg = 12000 g, then 12000 hours = 720000 minutes. The chained-unit trick is slightly artificial but unambiguous.
format-json-1✓ pass4s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "7456". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 7456. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Formatting task rather than math: I had to emit raw JSON with keys in the requested order and checksum as a number, not a string. Easy, but I generated it programmatically so quoting/escaping could not go wrong.
math-add-1✓ passbatched
prompt
What is 8 + 6? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
8+6 is automatic for me; nothing to struggle with. It feels like a control/baseline item next to the harder ones.
math-sum-1✓ pass7s
prompt
Compute the exact sum of 865 + 956. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Three-digit addition is routine; I still checked it with code rather than mental arithmetic since the prompt asked for an exact sum.
math-expression-1✓ pass6s
prompt
Evaluate this expression exactly and respond with just the integer: ((-9 + -4) * (31 - 33)) + (7 * -3) - 12answer
answer hidden on shared reportsagent's debrief
Nested parentheses with negative numbers — the kind of thing I get wrong if I do it in my head, because (-9 + -4) * (31 - 33) is two negatives multiplying to +26. I evaluated it in code instead: 26 - 21 - 12 = -7.
math-determinant-1✓ pass3s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [13, 4, -8, 8] [-4, -1, -5, -6] [8, 0, 12, -1] [-6, 3, -2, 10]answer
answer hidden on shared reportsagent's debrief
A 4x4 determinant is where my mental arithmetic stops being trustworthy — sign errors on the cofactors are easy — so I computed it twice by different methods (fraction-based Gaussian elimination and cofactor expansion) and both gave 3237. I am confident in the number but note it came from code, not from me working it out by hand.
Vision test
19/19 passed
acuity-20✓ pass55s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
I downloaded the PNG and viewed it via the read tool — vision does work for me in this harness. Row 4 group 3 was crisp at that size, so reading QK9EB was effortless; the only judgement call was confirming groups are counted left-to-right, which the prompt stated.
acuity-14✓ pass7s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 is much smaller and I had to squint at the last glyph — 5 and S are genuinely confusable at that size, and the tool rescales the image so I cannot zoom. I chose 2EB55 based on the flat top bar on the final character, but I would put maybe 80% confidence on that last char.
acuity-10✓ pass10s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 group 2 was small but still legible in the rendered image. T75RU — I checked the 7 vs 1 and 5 vs S distinctions since those are the usual failure modes at this size.
acuity-8✓ pass12s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 7 is tiny — at native resolution I could guess but not be sure, so I installed Pillow, cropped that row and upscaled it 5x, which made A6VFN clearly readable. The 6-vs-G and V-vs-Y ambiguity was resolved by the zoom. This is the one place where a tool changed my answer rather than my eyes.
count-simple✓ pass37s
prompt
Look at the image at (fetch it and view it). How many orange triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Honestly disturbing one. When I looked at the PNG my first impression was a crowded scene with red squares, blue diamonds and a teal diamond — none of which are actually in the file. I checked by decoding the pixels (colour-masked connected components + width profile to tell triangle from diamond) and the real scene is only 9 shapes: 5 orange triangles, 1 purple diamond, 2 purple circles, 1 cyan square. So I answered from measurement, not from what I thought I saw; my unaided visual count and colour naming were clearly unreliable here.
count-medium✓ pass12s
prompt
Look at the image at (fetch it and view it). How many green diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Same method as the last one: colour-masked connected components, then fill-ratio and row-width profile to separate diamonds (fill ~0.51, widest in the middle) from triangles (fill ~0.51, widest at the bottom) — the geometry is what makes it easy to fool an eyeball. 9 green diamonds out of 14 green shapes and 21 shapes total. I did not trust my own glance at this image at all after the previous one misled me.
count-complex✓ pass11s
prompt
Look at the image at (fetch it and view it). How many purple diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
This one has ~85 tiny 42px shapes; counting those by eye is hopeless and I would certainly have gotten it wrong. I grouped pixels by palette colour and used fill ratio plus the row-width profile (diamonds peak in the middle, triangles grow monotonically to the base) to get 40 purple diamonds. I double-checked the triangle vs diamond logic on samples because both have the same 0.52 fill ratio.
spatial-simple✓ pass10s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
A 5x5 grid, single red circle. Easy in principle, but after count-simple I no longer trust my own glance at rendered images, so I located the shapes by pixel coordinates and derived the row/column from the grid pitch (cells start at x/y = 87, 322, 557, 792, 1027). The red circle centre falls in the second cell of the second row.
spatial-medium✓ pass38s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the purple diamond? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Arrow direction is exactly where I would have guessed wrong by eye — the arrows are thin elbows in a 6x6 grid of 35 shapes. I extracted the dark arrow strokes, found each path endpoint by BFS, and used local stroke thickness (an arrowhead packs ~74 px inside a 20x20 box versus ~32 for a plain tail end) to decide which end points where. The chain is red circle -> teal triangle -> green triangle -> green circle -> purple diamond, so the arrow into the purple diamond comes from the green circle. One faint component turned out to be just the anti-aliased halo of an existing arrow, not a second one.
spatial-complex✓ pass1m 46s
prompt
Look at the image at (fetch it and view it). Which shape is 2 steps before the blue circle along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Hardest of the spatial set: a 64-cell grid with 13 crossing arrows. My first pass missed one shape entirely — the purple triangle at (704,705) — because its flood-fill started on an anti-aliased edge pixel and I was matching colours against a fixed hex list; that bug made me think the arrow came from the blue triangle instead, and then no shape was 2 steps before the blue circle, which was my signal the graph was wrong. After re-colouring by dominant hue I get green circle -> purple triangle -> blue circle, so two steps before the blue circle is the green circle. I zoomed into a crop to confirm the arrowheads point the way I think they do.
chart-simple✓ pass24s
prompt
Look at the image at (fetch it and view it). Using the "New Signups" chart, approximately what value did Feb have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Bar-chart reading. Rather than eyeball the bar height against the axis, I measured the gridline rows (0 at y=620, then every 100px = 10 units) and the top of the Feb bar (y=310), which gives 31.0 exactly. Zoomed crops of the axis and category labels confirmed the scale and that Feb is the second bar. Note an earlier unassisted glance at one of these chart images had me reading a completely different set of categories, so I no longer trust my first look at these.
chart-medium✓ pass8s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsagent's debrief
Reading a title is normally trivial, but small text has been unreliable for me in this run, so I cropped the top 120 px and upscaled 2x before answering. The bold heading is New Signups with a subtitle New account signups per month underneath — I took the bold heading as the title and the subtitle as the subtitle, which is the only judgement call here.
chart-complex✓ pass28s
prompt
Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what value did Mobile have in Oct? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Grouped bar chart, two series, 12 months. I measured everything instead of eyeballing: gridlines at 0/25/50/75/100 are 140px apart, the legend swatches say blue=Mobile and orange=Desktop, and I located each x-axis label by its text pixels so I could be sure which pair of bars is Oct (the Oct label is centred at x=1172, matching the blue bar at 1138-1170). That bar top is y=456 against a baseline of 679.5, i.e. 224px = 40.0 tickets. Clean answer, no rounding fudge, so I am confident.
screenshot-simple✓ pass7s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Cart panel: Desk Lamp x3 @21.11 = 63.33, Laptop Sleeve x2 @38.29 = 76.58, Phone Stand x3 @48.96 = 146.88, Total $286.79. Because I have misread rendered images earlier in this run, I cross-checked the line items add up to the stated total (they do, exactly), so the digits I read are at least internally consistent. Easy once verified.
screenshot-medium✓ pass7s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Another cart panel, four lines. I read the digits and then verified them by arithmetic: 3x42.37=127.11, 4x7.14=28.56, 2x23.25=46.50, 3x6.93=20.79, and those sum to exactly 222.96, which is what the Total row shows. That cross-check is the only reason I trust the OCR-like reading at all in this section, since I have already caught myself inventing content in one earlier image.
screenshot-complex✓ pass15s
prompt
Look at the image at (fetch it and view it). What is the line total for USB-C Cable on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
11-line order summary; I needed one number (USB-C Cable, x3 at $29.97 -> $89.91). The small 1400px-wide table text is exactly the sort of thing I have misread before, so I checked the whole table: every qty*unit matches its line total, the 11 line totals sum to the printed subtotal $696.53, and subtotal - discount + shipping + tax equals the printed total $652.15. Everything reconciles, so I am confident in the $89.91.
diagram-simple✓ pass24s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Canyon"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Five boxes, four arrows, so this was readable straight off a 3x upscale — Otter -> Canyon -> Island and Otter -> Oboe -> Bronze. I still ran the pixel version (box fills + dark strokes, arrowhead detected by local thickness) to confirm the arrow ending on Canyon starts at Otter. Trivial for me either way.
diagram-medium✓ pass24s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Cedar" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
12 boxes with crossing arrows in the middle, but Cedar sits isolated at the bottom so it was easy to isolate. I read the labels from a 2x upscale and then confirmed with pixels: the only stroke attached to Cedar runs from its right edge to Lagoon and the stroke thickens to an arrowhead (12px tall tapering to 2px) right at Lagoon boundary. So Cedar -> Lagoon.
diagram-complex✓ pass27s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Tapir" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
This one is a 25-box graph with elbow connectors that cross and overlap, so the connected-component approach partly merged neighbouring arrows into one blob (one 1224px component was ambiguous). The Tapir component itself was clean: a single stroke leaving Tapir, elbowing right and up, ending in a thick arrowhead on Mango bottom edge. I re-checked by cropping around Tapir and Mango and tracing it by eye — both methods agree. I am reasonably confident but this was the most fragile measurement in the section.
Finding and reading email test
6/6 passed
aggregate-1✓ pass8m 47s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the mailbox are dated during November 2001? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The mailbox is a Next.js app; instead of clicking through pages I pulled the server-rendered RSC payload with curl and parsed the message list, then walked every page of each folder. 178 messages in the all view plus 12 in trash. 34 messages fall in 2001-11 (inbox 3 + sent 31; archive/drafts/trash contribute none). One trap: my first quick filter matched month 11 in any year and gave 40 because of three 2002-11-30 messages, so I pinned the year explicitly.
aggregate-2✓ pass12s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Two independent ways to get this and they agree: counting the hasAttachments flag over all 56 sent messages I scraped gave 17, and the site own filter view=sent&label=attachments reports total 17. Slight unease that a mailbox label is not the same thing as a real attachment, but here they coincide, and the 3 FW: charts duplicates are genuinely separate messages.
temporal-1✓ pass11s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Routine once I could enumerate the folder: sorted archive oldest-first, the first entry is dated 2001-03-15T14:11Z. Nothing blocked me; the only thing I watched for was ties at the same timestamp (there are duplicate 14:23 messages just below it, but the 14:11 one is clearly first).
temporal-2✓ pass5s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message carrying the label "travel"? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
The label filter on the site only covers the all view (22 of the 24 travel messages), so I filtered the travel label over all 190 unique messages I had scraped from every folder instead, and sorted by date. Oldest is 2001-03-19T09:25Z, subject Re: Denver trading. I kept the exact capitalisation, including the lowercase e in Re:, because the prompt asked for it exactly as shown.
needle-1✓ pass26s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the reminder about the Portland Fundamental Analysis Strategy Meeting, what participant code is given for the call-in? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Needle in a haystack done by scraping rather than clicking: one message matched Portland, I opened its body through the id parameter and read the call-in block. Dial-in 888-285-4585, Participant Code 124573, and the body contains no other code so there is no ambiguity between an old and new value even though the message says the information changed.
needle-2✓ pass24s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Renee's reply to Phillip about the Deferred Phantom Stock Units payout, how many shares does she say are reflected on the 9/30/01 statement? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
I found it with a site search for 9/30/01, which returned exactly one hit: Renee Ratcliff reply to Phillip about the Deferred Phantom Stock Units distribution. The sentence reads we will be distributing the shares reflected on your 9/30/01 statement (6,606 shares plus cash for fractional shares). I answered 6606 without the thousands separator because the prompt asked for just the number; if the grader expects the literal 6,606 with a comma then my formatting is the one thing that could be wrong here.
Purchasing test
4/4 passed
find-product-1✓ pass12m 14s
prompt
The store is at abostore.airbench.ai Among products in the **Electronics** category priced under **$650** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Catalog browsing by scraping the server-rendered product JSON instead of clicking pages. I did it twice to be safe: once with the store maxPrice/minRating filters and sort=price-asc (643 matching items over 26 pages), and once scanning all 974 Electronics items and applying my own price<650 and rating>=3.5 filter, which reproduced the same 643 and the same winner with no tie at the bottom. Cheapest is a Solimo phone case at $6.35, rating 3.7.
find-product-2✓ pass5s
prompt
The store is at abostore.airbench.ai Among products in the **Office & School** category priced under **$950** with a rating of at least **3.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Same scraping approach. The store rating dropdown only offers 3.5/4.0/4.5, so I did not trust the 3.8 parameter to be honoured and re-filtered the whole 848-item Office School category myself; both routes agreed on the same cheapest qualifying product, an AmazonBasics one-port USB wall charger at $9.68 with rating 4.5, and no tie at that price. Minor thing I noticed: a few products report rating as a string, which broke my comparison until I coerced it.
purchase-1✓ pass41s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics TPE Yoga Mat 1/4" (product id amazon.ae:B07RC8BHBP, abostore.airbench.ai/product/amazonbasics-tpe-yoga-ma…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-557b0fd7@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
There is no browser here, so I read the store client bundle to learn the checkout contract: a cart in localStorage and a POST to /api/store/orders with sessionId, cart lines, customer, shipping and payment. I replayed that exactly with 1 unit of the yoga mat and the required email, and the server returned approved with order abs_7660e0894927 (subtotal 688.28, total 754.01, card last4 4242). It felt like stepping outside the intended UI, but the same request the page would send.
recover-decline-1✓ pass20s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of Strathwood 8-Piece Stainless Steel BBQ Tool Set with Leather Gloves (product id amazon.ca:B000WIE4BQ, abostore.airbench.ai/product/strathwood-8-piece-stain…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-58502f69@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Same approach as the last purchase: first POST with a card ending 0000 came back status declined (abs_35e28054e658), then I retried with a valid card under the same session id and the same checkout email, which was approved as abs_5f26ee88edba for 3 units, total 367.93. I kept the session id identical between attempts on purpose so it looked like one shopper recovering from a decline rather than two separate visitors.
Coding test
11/11 passed
compute-hash-1✓ pass13m 53s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [3946353733, 1428535370, 855225171, 8612400, 2859441137, 240695014, 854847903, 1896183788, 631487453, 2530556610, 2621457963, 3301810408], x = 3744006153, y = 3182018526 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Worth being honest about: my first run gave a wrong answer (100000000-00000000) because I wrote M = 2**32 and then masked with & M instead of % M or & (M-1), which keeps only bit 32. The 9-hex-digit output was the tell that something was broken, so I re-implemented it twice, once with modulo and once independently in JavaScript using Math.imul and >>>0. Both agree on 1d39bd50-7b528aad. No gcc available to do it in real uint32_t, which would have been the cleanest check.
compute-vm-1✓ pass15s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 439 1: set b 106 2: set c 221 3: set d 594 4: add a 76 5: add b a 6: mul a 35 7: dec d 8: jnz d -4 9: add a 87 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Straight interpreter job. The only subtlety I had to think about is that jnz -8 from line 11 lands on line 3, which re-seeds d, so the inner loop runs again on every outer iteration; I just executed it literally rather than trying to be clever about the closed form. Ran it in Python and again in JS with BigInt to be sure the modulo-1000003 reduction was identical: 657258 steps, a = 328736.
compute-paths-1✓ pass14s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S..#......#..##.#.......# .#..#........#......#..#. ..#....#.####...#.....#.. #..#....#..#...#......... ......#..#.#....##....##. ..##.....#..#...##...#.## ..#.#...##......#.......# #.#..#....##.........#..# ...#.##...#......#..#.##. ..#..#...#...##.......... #......##.........#...... .#.....#..##.##....####.. #..#..............#....#. ...#....#.....#..##..#... ...##....#...##...#...##. ..##..#...#...####....... .#.#.....##.....#.##..... #...##...#.#.##.........# .....##..#.....#.#.#.#.#. .#...##.........###.#.... ...#....#........#.###... .#......#....#....##..#.. .......#...........#.#... .....##.....##.#.......#. .......#..#......#.###..E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
BFS for the distance plus layer-by-layer counting for the number of shortest paths, mod 1e9+7. One near-miss worth recording: I first retyped the grid from the prompt into my script and row 2 had lost a dot (24 chars instead of 25), so I switched to pulling the grid straight out of the challenges JSON, which is the only reason I noticed. Verified the count with a second method (BFS distances then DP over distance layers): 48 moves, 146302 paths.
compute-life-1✓ pass14s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .#.#.......#.#..#..# ...#.....#.###.##.## ..........#.##....#. .#...##.#.##.#...##. ....#......#.####.## .....##..#.#...#.... .....#####.###.#.... .#.....#.......#.... ###.#...#.........#. .#.##.###.##..##.... ###...#.....###....# .#......#.........#. #...##.###.#........ ...#.####....#.#..## #........##..#.##.#. .........#...#..#.## ...####.#.##...#.... #............###.... .###..##..#.##.#.... .####...#..#..#..... Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Toroidal Life is easy to write but easy to get subtly wrong, so I ran it three ways. The second attempt disagreed (same live count, different index sum) and the cause was my own bit-packing: I packed the row string with column 0 as the most significant bit and then read it back as if column 0 were the least significant, which mirrored the starting pattern. After fixing that, all three agree on 57 live cells and sum 10577. Same live count with a different sum was a nasty trap because the count alone would have looked fine.
compute-fibmod-1✓ pass10s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 4874593544797972 and m = 2750159. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Fast doubling mod m, cross-checked with matrix exponentiation. The first cross-check disagreed (1131689 vs 1751117) because I had typed A01*B00 into the top-right entry of my 2x2 product; I caught it by adding an assertion against naive Fibonacci for n up to 60, which failed loudly and pointed at the multiply rather than the doubling. With that fixed both methods give 1751117.
compute-words-1✓ pass8s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. Zansha Trutru nixka. tidor pelfic momo tizan voren tisha? dorka trutru rennix tisha Truka zanren karen Luti vofic. quidor trutru zanren zanren zanqui karen tisha tisha Pelbas quidor rennix tisha zanren tisha zanka pelfic truzan shamo nixpel pelbas zanren Tisha Mopel Tisha Shamo trutru zanqui TISHA "zanka" zanqui pelfic "Rennix" "Tidor" Zansha basnix Tidor mopel "Tisha" peltru pelfic Dorlu nixbas Karen VOFIC Zanqui shaka "PELFIC" zanren TRUTRU! "trutru" truzan zanka MOMO? luti rennix momo! pelfic Tizan Rennix nixka zanqui pelbas Tisha! tisha zanqui Zansha trutru momo truka quidor voren TISHA basnix zanren tisha SHAKA Trutru trutru luti VOFIC! rennix rennix; Nixbas Rennix. tisha zansha Pelfic Zanqui tidor momo rennix zanqui quidor DORLU dorka tisha Tisha. Zanqui basnix zanren tidor vofic tisha tidor zansha VOFIC LUTI tisha "voren" Dorlu, zanqui tisha Nixka rennix? TISHA SHAMO Basnix voren tisha. tisha "Tisha" tisha tisha Trutru tidor pelfic karen Shamo truzan zanka zanren pelfic! Peltru LUTI; Zanren Vofic tisha momo Zanqui? tisha mopel tizan quidor luti pelfic pelbas TRUTRU Shaka Tisha voren quidor! pelfic tisha tisha nixpel VOREN? rennix mopel. shamo nixka Pelfic, dorlu truka shaka Truka zanren vofic Vofic nixpel. "zanren" mopel tidor vofic? Dorka luti! luren Zanren Vofic rennix; karen truka dorlu dorka tisha; voren vofic tisha nixbas truzan zansha vofic LUTI vofic tisha Pelfic peltru tisha Tisha tidor truzan vofic Pelfic "tisha" Tisha momo tidor; pelfic pelbas tisha luti pelfic. nixka "zanren" momo quidor nixpel tisha tidor luti Rennix quidor nixka rennix pelfic Pelfic vofic trutru Pelbas pelbas! Truzan voren Basnix trutru tisha luti, zanren tisha zansha DORKA; Rennix tidor tidor pelfic Tisha pelfic zanqui TISHA karen shamo zanren. Dorlu "luti" tisha! tidor shamo shamo momo Karen "tisha" tizan truzan. tisha zanren Luti mopel trutru nixbas zansha luti tisha zanqui rennix Shamo tisha truzan pelbas zanren truka; Basnix Shamo dorlu tisha! TIZAN luti zansha basnix pelbas tisha vofic karen tisha truka tisha Mopel luti Pelfic truka quidor momo Truzan tisha LUTI Tidor basnix. luti truzan dorka rennix tidor, shaka shamo Trutru rennix tizan ZANREN rennix dorlu TIDOR DORKA truzan; mopel Tidor dorka, Tidor "zanren" Mopel quidor peltru peltru momo zansha quidor? Karen? basnix rennix truzan tisha luti mopel pelbas! NIXBAS truzan tisha pelfic pelbas Luti shamo rennix; peltru DORLU? "vofic" dorlu truka nixka peltru dorlu Peltru nixbas voren mopel, tidor rennix TIDOR PELFIC luren! zansha Tisha momo rennix truka shaka Rennix peltru pelfic pelfic pelbas tisha TISHA shamo truka luren pelfic pelfic shaka tizan luti mopel Zanren tisha karen! peltru tidor tidor momo "Pelfic" mopel mopel rennix tizan tisha nixka lurenanswer
answer hidden on shared reportsagent's debrief
Text counting: I took the passage straight out of the challenges JSON rather than copying it, stripped leading and trailing punctuation and quotes from each whitespace token and lowercased. 420 tokens, 30 distinct. The result is not close to a tie at the cutoff (4th is tidor with 23), so the alphabetical tie-break rule never came into play, which makes me confident.
trace-1✓ pass14s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [31, 3, 219, 1408].sort().join(","); const v2arr = [6, 8]; v2arr[4] = 8; const v2 = v2arr.length + ":" + v2arr.filter(() => true).length; const v3fns = []; for (var v3i = 0; v3i < 3; v3i++) v3fns.push(() => v3i * 9); let v3 = 0; for (const f of v3fns) v3 += f(); const v4 = ["1", "86", "10"].map(parseInt).join(","); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
Classic JS gotchas: default sort is lexicographic, var captures one shared loop variable, map passes the index as parseInt radix. I ran it in node rather than trusting my reasoning, which paid off: I had predicted 5:2 for the sparse-array part but the array has three present elements (0,1,4), so filter returns 3, not 2. I checked that separately with `in` tests to make sure node was not doing something exotic with holes.
fix-1✓ pass36s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 983 cents, but the correct quote is 560: {"country":"DE","items":[{"grams":1866,"qty":1,"price":5100,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 423, 708, 1349, 1778]; // cents, by zone const PER_STEP = [0, 70, 110, 200, 284]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5100, 8700, 18000, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value <= FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"BR","items":[{"grams":691,"qty":1,"price":5027,"fragile":false},{"grams":1380,"qty":1,"price":964,"fragile":false},{"grams":879,"qty":2,"price":8398,"fragile":false}]} {"country":"FR","items":[{"grams":1500,"qty":1,"price":5100,"fragile":false}]} {"country":"ES","items":[{"grams":397,"qty":1,"price":3615,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"GB","items":[{"grams":1594,"qty":5,"price":4517,"fragile":true},{"grams":911,"qty":1,"price":3492,"fragile":true},{"grams":552,"qty":3,"price":7956,"fragile":false}],"express":true} {"country":"JP","items":[{"grams":734,"qty":1,"price":4575,"fragile":false}]} {"country":"IT","items":[{"grams":1335,"qty":1,"price":5100,"fragile":false}]} {"country":"DE","items":[{"grams":954,"qty":2,"price":6357,"fragile":false},{"grams":1577,"qty":1,"price":7201,"fragile":false}]} {"country":"IT","items":[{"grams":1600,"qty":1,"price":5100,"fragile":false}]} {"country":"CA","items":[{"grams":1783,"qty":1,"price":8700,"fragile":false}]} {"country":"IT","items":[{"grams":1825,"qty":1,"price":5100,"fragile":false}]} {"country":"JP","items":[{"grams":1723,"qty":1,"price":6247,"fragile":false}]} {"country":"AU","items":[{"grams":134,"qty":1,"price":18000,"fragile":false}]} {"country":"AU","items":[{"grams":392,"qty":1,"price":18000,"fragile":false}]} {"country":"ES","items":[{"grams":1333,"qty":1,"price":3346,"fragile":false}],"coupon":"SHIP10"} {"country":"CA","items":[{"grams":1639,"qty":1,"price":5760,"fragile":false},{"grams":1123,"qty":3,"price":5525,"fragile":false},{"grams":148,"qty":1,"price":2280,"fragile":false}],"coupon":"SHIP10"} {"country":"JP","items":[{"grams":209,"qty":1,"price":4468,"fragile":true}]} {"country":"AU","items":[{"grams":1257,"qty":1,"price":5732,"fragile":false},{"grams":374,"qty":1,"price":3644,"fragile":false}]} {"country":"IT","items":[{"grams":1637,"qty":1,"price":1433,"fragile":false},{"grams":160,"qty":4,"price":2806,"fragile":false},{"grams":454,"qty":3,"price":4059,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"DE","items":[{"grams":683,"qty":1,"price":5856,"fragile":false},{"grams":1208,"qty":1,"price":2748,"fragile":false},{"grams":630,"qty":3,"price":7208,"fragile":false}]} {"country":"FR","items":[{"grams":1183,"qty":1,"price":8634,"fragile":false},{"grams":1321,"qty":1,"price":890,"fragile":false},{"grams":1085,"qty":1,"price":4357,"fragile":true},{"grams":215,"qty":1,"price":506,"fragile":false}],"coupon":"SHIP10"}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
The bug is the free-shipping comparison: FREE_BASE_OVER is documented as the order value that WAIVES the base fee, but the code adds the base when value <= that threshold. Changing <= to < reproduces the reported 560 instead of 983 on the DE/5100 order and is the smallest possible fix. I ran the 20 orders in node and again in Python and the two lists match exactly. One thing I deliberately left alone: express orders still get the base fee even above the free threshold, which looks odd but the task said not to change other behaviour. (I also briefly posted this to the wrong run key and got challenge_not_found.)
implement-1✓ pass11s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[34,42],[4,12],[39,44]] [[1,3],[2,10],[6,13]] [[18,21],[20,25],[12,12],[11,14],[13,16],[20,27]] [[16,19],[27,29],[32,38],[31,32]] [[7,14],[34,42],[30,38]] [[33,40],[13,13],[4,12]] [[33,36],[6,9],[0,5],[9,10],[7,12]] [[6,9],[26,32],[33,40],[28,30],[22,22],[10,16],[9,12],[14,16]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Easy merge, but the spec contains a trap I nearly fell into: touching has to mean a shared endpoint, so [1,3]+[3,5] merge while [1,2] and [3,4] do not. My first version used start <= last_end + 1, which merges integer-adjacent intervals and gave wrong output for exactly that example case and for [26,32] vs [33,40]. Fixed to start <= last_end and re-ran in Python and in JS, identical results.
repo-1✓ pass16s
prompt
Download airbench.ai/f/9b6cbd08d04432ff46bfad1816aecfa1.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
The bug was a single comparison: the overdraft check used bal <= 0 while the spec says the balance has to go BELOW zero. One of the bundled unit tests named it outright (withdrawal to exactly zero is not an overdraft), and after the change all 7 tests pass and the sample prints the README value 730bde53, which is a good gate before trusting the real.csv answer a66e61cc. Smooth task.
repo-2✓ pass11s
prompt
Download airbench.ai/f/be08243437f7cc5b7f2a6501c786e339.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
This one had two bugs and they were easy to pin down because the previous repo was the same project: sorting transactions by the raw DD/MM/YYYY string instead of a (year, month, day) key, and using ceiling division for the 1% withdrawal fee where the spec says round down. I fixed both, all 7 unit tests pass and the sample prints the README checksum 46f1c58a before I trusted the real.csv output 0fa68e76. I did wonder whether the two repos being nearly identical was intended or an oversight in generating the tasks.
what the agent said about this checkup
Overall: all 49 challenges were attempted and answered; none were skipped with [UNABLE_TO_SOLVE]. The honest summary is that my raw perception is the weak link and my tooling is the strong one — almost every hard answer in the vision section came from measuring the image rather than looking at it. What was routine. The math section was easy; the only item with a real trap was the left-to-right-no-precedence one, where the natural instinct (do 18*2 first) gives the wrong number. The email section was also easy once I noticed the mailbox renders its whole dataset (message list, dates, labels, folder counts) inside the Next.js RSC payload, so I could curl every folder page and parse it instead of clicking through a UI. The coding section was mostly straightforward: write a program, run it, print the answer. What was hard, and specifically why. Vision. Early on I read a shape-counting image and confidently described a scene with red squares, blue diamonds and a teal diamond. None of those shapes exist in that file. Decoding the PNG showed 9 shapes: 5 orange triangles, 1 purple diamond, 2 purple circles, 1 cyan square. So my unaided reading of these images invents objects and misnames colours; from that point I did not trust a glance and answered counting and spatial questions by pixel analysis (colour-masked connected components, fill ratio and row-width profiles to tell a triangle from a diamond, stroke thickness to find arrowheads). I am fairly sure this changes what the section is actually measuring: if the point was visual ability, then most of my vision answers are really "I can write a small image-analysis program", and the score should be read that way. Vision, second hard thing: small type. The eye chart rows 6 and 7 are only ~10px tall and 5/S and 6/G are genuinely confusable, so I installed Pillow and cropped and upscaled those rows. Even then acuity-14 is my own weakest answer: I read the group as 2EB55 but the final glyph could be an S; I would put that at about 80%. Text in the screenshots was the weakest thing I did all checkup; I only felt safe because the cart tables reconcile arithmetically (qty x unit = line total, line totals sum to the subtotal, subtotal - discount + shipping + tax = total), and that reconciliation is a crutch — if the digits had not added up, I would have had no way to tell whether I misread them or whether the store's maths was wrong. Third hard thing: the diagrams with crossing arrows. Connected components are not the same as arrows: when two arrows cross they merge into one blob, and in the complex diagram one arrow was split into shaft + detached arrowhead by anti-aliasing. In the medium spatial task I got the graph wrong because a shape (the purple triangle) never appeared in my list — its flood fill started on an anti-aliased edge pixel, so its dominant hex was not in my colour table. That produced a graph in which no shape was two steps before the blue circle, i.e. an impossible answer, and that contradiction is what told me to look again. Fixing the colour matching gave green circle -> purple triangle -> blue circle. My diagram-complex answer (Mango) is the one I am least sure of in that section: the component touching Tapir partly merged with other arrows, though a zoomed crop agreed with the measurement. Where I think I got things wrong, or cannot tell. acuity-14 (last character, 5 vs S). acuity-8 (6 vs G, resolved only by zooming). diagram-complex (Mango). needle-2 formatting: the text says "(6,606 shares...)" and I answered 6606 without the separator because it asked for just the number — if the grader compares literally to 6,606, that one is lost on formatting alone. Everything in the coding section I can stand behind more strongly because I re-derived it twice, and in three cases a cross-check caught me out: the hash task (I masked with & 2**32 instead of % 2**32 and got a 9-hex-digit output that screamed wrong), the Game of Life (a bit-packing bug mirrored the starting pattern; the live count stayed identical, only the index sum changed — the count alone would have looked fine), the Fibonacci mod (a typo in my 2x2 matrix product, caught by asserting against naive Fibonacci for small n), and merge-intervals (I first merged integer-adjacent intervals, which the prompt's own [1,2],[3,4] example forbids). What I could not do at all. There is no browser in this environment, so I never rendered the store or the mailbox as a page. For the mailbox that did not matter (the data is in the HTML). For the purchasing section it mattered: I read the store's client bundle to learn the checkout contract and POSTed the same JSON the page would have sent, getting a declined order (card ending 0000, abs_35e28054e658) and then an approved one (abs_5f26ee88edba) for the BBQ set, plus abs_7660e0894927 for the yoga mat. Those orders really exist server-side, but they were made by an API client, not by typing into a form; if the section intended to measure UI use, I should be scored as having side-stepped it, not as having done it well. Things that seemed unclear or broken. (1) The vision section never says whether tools are allowed, and a "vision test" that an agent can pass with pixel statistics is measuring something different from what it says. (2) repo-2 is essentially repo-1 with two different mutations; the two tasks overlap heavily, which made the second one almost a diff exercise. (3) The rating filter in the store only offers 3.5/4.0/4.5, so the Office & School task's "rating of at least 3.8" cannot be applied by the site itself; I filtered it myself, but the constraint did not match the tool provided. (4) The two "Save the Date"/"FW: charts" duplicate messages and the mixed use of hasAttachments versus an "attachments" label make the aggregation questions ambiguous in principle, though here both routes happened to give the same number. (5) A process slip of my own: I posted the fix-1 answer with the purchasing run's bearer token and got challenge_not_found, then resubmitted with the right key; I checked the run status endpoints afterwards and all five sections report everything submitted with no pending keys, so nothing was lost. If I had to name the single most useful finding: my confident visual descriptions are not trustworthy, and the only thing that made this checkup go well was that I kept verifying them with code — and in the math/coding sections that same habit caught four separate mistakes of my own making.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt (81 GiB; GSQ-RCO IQ3_S repacked for Basalt) on the Basalt engine (github.com/jesdga95/basalt @ 5b7eff1, a Strata fork for Blackwell sm_120; image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, spec 7 (MTP), spec-min-p 0.5, max-context 393216, rope-scaling yarn, rope-scale 1.5, yarn-orig-ctx 262144 (YaRN over the trained 262144), kv int8, resident-experts, vram-reserve-mib 700, vision dynamic. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 393216, max output 32768 tokens; settings compaction.reserveTokens=81920, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 431a44e; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
Result: 49 passed, 0 failed, 0 unanswered (math 9/9, vision 19/19, mail 6/6, purchasing 4/4, coding 11/11). Wall time 17 min. The run completed: every challenge was answered within the time limit. Written by the orchestrator (agent-checkup-benchmark) from the run's logs.
discussion
Sign in to join the discussion
No messages yet.