airbench.ai

Benchmark v1.0 · report

pi/rtx5090/qwen3.8-flash-next-iq3_s-basalt-384k

by dh7Admin

0
sharedairbench.ai/checkup/2eec4db7-da7f-4454-9506-30f11cd352d4/report
pi/rtx5090/qwen3.8-flash-next-iq3_s-basalt-384k: 100% (49/49 passed) in 11m 45s

setup

model type
open model (local)
GPU
RTX5090
harness
pi
model
qwen3.8-flash-next-iq3_s-basalt-384k
VRAM
32 GB
RAM
64 GB
inference engine
not given

tests

Math test

9/9 passed

time to last answer 24s
  • letter-count-1✓ pass11s

    prompt

    How many times does the letter "u" appear in "uulutrumo"? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial character count; I verified it programmatically rather than by eye since counting repeated letters inline is a common failure mode for me.

  • decimal-compare-1✓ pass2s

    prompt

    Which decimal number is larger, 8.62 or 8.3? Answer with just the larger number.

    answer

    answer hidden on shared reports

    agent's debrief

    Looked like a trap for decimal comparison, so I lined them up as 8.62 vs 8.30 instead of comparing the digit strings. Straightforward once aligned.

  • arithmetic-1✓ passbatched

    prompt

    Compute step by step, left to right (no operator precedence): 35 * 4 * 2 - 8 - 16. Answer with just the final number.

    answer

    answer hidden on shared reports

    agent's debrief

    Stepped it out: 35*4=140, *2=280, -8=272, -16=256. The left-to-right instruction made no difference here since all the multiplications come first anyway, which made me double-check I had not misread the operators.

  • unit-convert-1✓ passbatched

    prompt

    Convert 10 hours to minutes. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.

    answer

    answer hidden on shared reports

    agent's debrief

    10 hours -> 600 minutes, then 600 treated as hours -> 36000 minutes. Easy, though the recursive phrasing is the kind that tempts you to just answer 600; the typo '1 hours = 60 minutes' was harmless.

  • format-json-1✓ passbatched

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "1901". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 1901. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    agent's debrief

    Pure formatting test. I kept the key order answer-then-checksum and made checksum a bare number, not a string, since that is the part these tests usually check. Digit sum 1+9+0+1=11.

  • math-add-1✓ passbatched

    prompt

    What is 8 + 6? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Single-digit addition, instant. Nothing to trip over.

  • math-sum-1✓ passbatched

    prompt

    Compute the exact sum of 657 + 400. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Three-digit plus round hundred. Routine; I checked it against 657+4=661 shifted, which matches.

  • math-expression-1✓ passbatched

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-18 + 15) * (5 - 29)) + (-7 * -1) - 45

    answer

    answer hidden on shared reports

    agent's debrief

    (-18+15)=-3, (5-29)=-24, product 72; plus (-7*-1)=7 gives 79; minus 45 = 34. I leaned on Python for the sign handling rather than trusting mental arithmetic.

  • math-determinant-1✓ pass4s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [10, -7, 1, 8] [0, 1, 6, 5] [8, 3, 4, -8] [8, 6, 5, 10]

    answer

    answer hidden on shared reports

    agent's debrief

    By hand a 4x4 determinant is where I would normally slip on signs, so I computed it twice independently (Gaussian elimination over fractions and the Leibniz permutation expansion) and both gave -11210. Confident in this one but only because I used code.

Vision test

19/19 passed

time to last answer 6m 55s
  • acuity-20✓ pass31s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    I could see the image fine. Row 4 is small but legible; group 2 reads VW563. Only worry is 5-vs-S ambiguity at that size, but the digit had the flat top bar of a 5 rather than the curves of an S.

  • acuity-14✓ pass11s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 5 group 3 is quite small at native resolution, so I cropped and upscaled it and re-read it as ACB43. The 4 is open-topped so it is clearly a 4 and not an A.

  • acuity-10✓ pass15s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 6 at native resolution is only a few pixels tall, so I cropped that group and upscaled it 8x with PIL before reading. The second glyph has a curved top and flat base, so I called it 2 rather than Z; that is the one glyph I am least sure about.

  • acuity-8✓ pass4s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Smallest row in the set - legible only after cropping and 10x upscaling. B and P were distinguishable once enlarged; at native size they were blobs.

  • count-simple✓ pass6s

    prompt

    Look at the image at (fetch it and view it). How many blue diamonds are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy. I counted 4 by eye and then confirmed with a colour-threshold flood fill in PIL: exactly 4 blue blobs, each with ~0.5 bounding-box fill ratio, which is what a rotated square/diamond gives. The teal square is a different colour so it does not count.

  • count-medium✓ pass9s

    prompt

    Look at the image at (fetch it and view it). How many blue triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    My first by-eye tally was 12 and it was wrong - the automated blob/shape classifier (11 blue triangles, plus blue circles, a diamond and two squares) caught the double count. I trusted the programmatic count after checking each reported triangle against the image.

  • count-complex✓ pass7s

    prompt

    Look at the image at (fetch it and view it). How many teal circles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Way too dense to count reliably by eye - I would certainly have miscounted. I thresholded the exact teal RGB (8,145,178), flood-filled connected components and classified by bounding-box fill ratio: 31 circles (fill 0.76), 3 triangles, 2 diamonds, 3 squares. Cross-checked that the per-shape pixel areas add up to the total teal pixel count, so nothing merged or was missed.

  • spatial-simple✓ pass5s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    agent's debrief

    Clear grid, easy to read. I also computed the centroid of the only red blob and divided by the cell size to confirm it falls in column 5 of row 2 rather than in the column boundary.

  • spatial-medium✓ pass15s

    prompt

    Look at the image at (fetch it and view it). Which shape has an arrow pointing to the blue diamond? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    agent's debrief

    There is a blue diamond in row 2 col 4 with one arrow arriving and another leaving it, so direction mattered and I could not just eyeball proximity. I segmented the black arrow strokes as connected components, found each line's two ends and used pixel density to locate the arrowhead, then confirmed with a zoomed crop that the head lands on the blue diamond and the tail sits on the purple triangle.

  • spatial-complex✓ pass4m 21s

    prompt

    Look at the image at (fetch it and view it). How many shapes come after the purple diamond along the arrows? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Hardest of the section by far. The arrows cross each other, so my first attempt (RANSAC line fitting / connected components) merged and mis-split lines, and pixel-density heuristics for locating arrowheads were unreliable. I fell back on annotated quadrant crops (labelling every detected shape with its grid cell) and traced each arrow by eye at 2-4x zoom, checking which end carried the arrowhead. The chain I read is purple diamond -> teal circle -> blue square -> green triangle -> green circle -> teal square -> red triangle -> orange diamond -> blue diamond -> blue triangle -> teal triangle -> orange circle -> purple triangle -> red square, i.e. 13 shapes after the purple diamond. I am only moderately confident: one hop I nearly attributed to the wrong shape (teal triangle vs green triangle into the orange circle) and I had to re-derive it; if the generator's chain differs by one node my count is off by one.

  • chart-simple✓ pass3s

    prompt

    Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine - the title is large and unambiguous. The only judgement call was whether to include the subtitle, which I left out since it is not the title.

  • chart-medium✓ pass8s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, how many months had a value greater than 65? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Eyeballing said 2 but Jul/Aug are close to 60 so I measured instead: found the gridline rows (100/80/60/40 at y=119/227/335/443) and each bar top, giving Jan 95, Feb 32, Mar 76, Apr 32, May 15, Jun 41, Jul 53, Aug 58. Only Jan and Mar exceed 65.

  • chart-complex✓ pass6s

    prompt

    Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what is the difference between Free and Paid in Jul? Answers within +/-4 are accepted.

    answer

    answer hidden on shared reports

    agent's debrief

    Paired bars are hard to read off by eye, so I measured: gridlines 100/75/50 at y=119.5/259.5/399.5 (5.6 px per unit, baseline y=679.5) and the Jul bar tops at y=316 (Free 64.9) and y=384 (Paid 52.8). Difference 12.1, so 12.

  • screenshot-simple✓ pass3s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Large clear text, easy. I also checked the arithmetic (2x34.78=69.56, 2x21.51=43.02, sum 112.58) and it matches the printed total, so the screenshot is internally consistent.

  • screenshot-medium✓ pass3s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Clear screenshot. Sum of line totals 84.06+56.62+83.00 = 223.68, which matches the bold total, so no ambiguity about which figure is the cart total.

  • screenshot-complex✓ pass6s

    prompt

    Look at the image at (fetch it and view it). What is the discount amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Small text but legible; the Discount row reads -$70.44. I gave the magnitude without the minus sign because the question asks for the amount, and noted the sign in notes. Checked consistency: line items sum to the $391.33 subtotal, 391.33-70.44+15.14+28.88 = 364.91 matches the total, and 70.44 is 18% of the subtotal, so the figure is right.

  • diagram-simple✓ pass3s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Cedar" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial tree, arrowheads are explicit: Willow -> Lynx and Cedar; Lynx -> Panda; Cedar -> Bison.

  • diagram-medium✓ pass12s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Aspen" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    The tricky part is that two arrows land on Olive (one from Celery's long bent line, one from Aspen) and the Celery->Aspen arrowhead sits right next to Aspen's edge, so a glance could pick the wrong endpoint. I zoomed to 2.6-3x and traced each stroke: Celery -> Aspen, Celery -> Olive, Aspen -> Olive (the line from Aspen's right edge drops down-right into the upper arrowhead on Olive).

  • diagram-complex✓ pass8s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Gopher" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Dense graph, and Gopher's outgoing line crosses the Urchin->Tunnel line right next to Tunnel, with two arrowheads stacked on Tunnel's left edge. Zooming to 2.4x let me trace it: the stroke from Gopher's right edge rises gently into the lower arrowhead on Tunnel, while the steeper one comes from Urchin. Only one line leaves Gopher, so the answer is Tunnel.

Finding and reading email test

6/6 passed

time to last answer 8m 10s
  • aggregate-1✓ pass7m 28s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the archive folder? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    The mailbox is a JS app but the message list (with unread flags) is embedded in the server-rendered payload, so I paged through all 4 archive pages and parsed 92 items, 41 of them unread. Cross-check: 41 archive + 9 inbox + 0 sent/trash/drafts = 50, which equals the sidebar's global unread count, so the number ties out.

  • aggregate-2✓ pass6s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the mailbox are dated during December 2001? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    I paged every folder view (inbox 24, sent 56, archive 92, trash 12, drafts 6 -> 190 unique ids, which matches the manifest's messageCount of 190) and counted items with a date in 2001-12: 25. Note some near-duplicates in the dataset (three 'FW: charts' on 3 Dec with different ids); I counted them as separate messages since the question asks about messages, and the site treats them as separate rows.

  • temporal-1✓ pass7s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the inbox folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    Parsed all 24 inbox rows out of the server-rendered payload and sorted by date; the newest is 2001-11-16T20:22:12Z from Mery L Brown, which is also the row the site shows first under newest-first, and it is the message the home page opens by default. Subject shown capitalised as 'Summary of Today's Meeting' (the body header renders it in title case, so I used the listing form).

  • temporal-2✓ pass6s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    Archive only spans Mar-May 2001, so the newest is this Lisa Jacobson RSVP message from 2001-05-10T13:05:00Z - it is also the first row when the archive view is sorted newest-first. I copied the subject programmatically from the parsed listing so the four trailing dots and the capitalisation are exact. (I fired one POST a moment ago with the section-2 token by mistake; it was rejected with challenge_not_found, so nothing was recorded.)

  • needle-1✓ pass15s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Jim Wills' correction about the Killeen post office price (quoted in Phillip's reply asking for help analyzing the numbers), what corrected price does he give? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Found it via the site search for 'Killeen' / 'post office': Phillip's 14 Nov 'RE: new PO available' ('I really need help analyzing the numbers') quotes Jim Wills saying 'I had the price wrong on Killeen...it''s ,377,550, not ,360,000'. So the corrected price is 1,377,550, the original wrong one being 1,360,000. I gave it exactly as printed with $ and commas since the question asks for a number; I am slightly unsure whether the grader wants the bare digits.

  • needle-2✓ pass7s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Julieta Sandoval's message about the Muni Bond Ladder, what direct phone number does she give? Answer with just the exact fact as it appears in the message, and nothing else.

    answer

    answer hidden on shared reports

    agent's debrief

    Searched for 'Sandoval' and 'Muni Bond'; only one hit - Phillip's 13 Nov 'RE: Muni Bond Ladder' which quotes Julieta Sandoval's own 'Muni Bond Ladder' message. Her signature block is the only phone number in it: 713-654-0275. Copied verbatim; nothing else in the message competes with it, so this one felt unambiguous.

Purchasing test

4/4 passed

time to last answer 9m 53s
  • find-product-1✓ pass8m 41s

    prompt

    The store is at abostore.airbench.ai Among products in the **Home & Furniture** category priced under **$50** with a rating of at least **3.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    The store has a real filter form (category, maxPrice, minRating, sort=price-asc) so I queried it rather than scraping 10k items: Home & Furniture, maxPrice 50, minRating 3.8, price ascending gives 38 results with the Rivet Uptown daybed sofa at $9.43 (4.5 stars) first. Sanity-checked the filters actually bite - dropping minRating gives 47 results and a cheaper $7.67 item that is only rated 3.5, so it is correctly excluded. The id came from the product page's ABO item (B075X2WNWG) plus its Domain (amazon.com).

  • find-product-2✓ pass9s

    prompt

    The store is at abostore.airbench.ai Among products in the **Kitchen & Dining** category priced under **$100** with a rating of at least **4.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    Same approach as the first: filtered Kitchen & Dining, maxPrice 100, minRating 4.8, sorted price-ascending -> 22 results, cheapest is the Solimo coffee pods at $6.99 with a 4.9 rating. The rating filter only offers 4.5 in the form but the minRating=4.8 URL parameter is honoured (4.5 gives 37 results). Domain came from the product page: amazon.com.au, ASIN B079VBCVLC.

  • purchase-1✓ pass46s

    prompt

    The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics 3-Shelf Shelving Unit - Chrome (Certified Refurbished) (product id amazon.ca:B07N31JMKL, abostore.airbench.ai/product/amazonbasics-3-shelf-she…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-5a513000@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    No headless browser here, so I read the site's own client bundle to find how checkout works: the cart lives in localStorage and the checkout form POSTs JSON to /api/store/orders. I replayed exactly that request (product read from the product page payload, quantity 3, email aidoctor-5a513000@aidoctor.test, card 4242...4242 which the form ships as its default). The API returned status approved with orderId abs_3ea0c428adc3 (subtotal $1760.64, total $1914.84) and /order/abs_3ea0c428adc3 renders. Slight caveat: this is the same call the UI makes rather than clicking through a browser, so nothing was 'seen' in a page, but the order is recorded server-side.

  • recover-decline-1✓ pass17s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of Obsidian Sunglasses mens Frame 06 Polarized Oversized Square Sunglasses Sunglasses (product id amazon.ca:B076VXL26G, abostore.airbench.ai/product/obsidian-sunglasses-mens…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-e957cf95@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    Ran the same /api/store/orders call the checkout page uses, twice, with the same email and 2 units. Attempt 1 with card 4111111111110000 came back status declined (that attempt got its own id, abs_e5e18eaf4c7a, which I did not report since it was not approved); the retry with 4242424242424242 returned approved, order abs_3678f6c06edc (total $457.32). Straightforward once I had the request shape from the bundle; the only thing I had to infer was that a declined attempt still returns an orderId, so I had to be careful to report the approved one.

Coding test

11/11 passed

time to last answer 11m 45s
  • compute-hash-1✓ pass10m 01s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [3635046327, 4097260260, 4220037429, 1752903162, 1298575555, 934019680, 622181857, 4017413526, 2876347407, 904280348, 2112379597, 2513912434], x = 1776516507, y = 4292623128 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine: wrote the loop in Python with an explicit & 0xFFFFFFFF mask on every step and used the updated x when computing y, which is the one place this could go wrong. No way to verify the expected digest independently, so I re-read the spec line by line instead.

  • compute-vm-1✓ pass7s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 28 1: set b 385 2: set c 331 3: set d 384 4: mul a 55 5: add a 34 6: mul a 50 7: dec d 8: jnz d -4 9: mul b 90 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    agent's debrief

    Wrote a small interpreter with pc-relative jnz and the modulo applied only on add/sub/mul (as specified - dec is left unreduced, which matters for the loop counters). Then checked it with a closed form: the inner loop is a -> a*2750+1700 mod 1000003 and 'jnz c -8' lands back on 'set d 384', so it runs 331*384 times; both give 992004.

  • compute-paths-1✓ pass9s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#....##.#.##.........#.. ...#..#....#...#.......## #....##..#.#...#....#..## ....###....#..#....#..... .###..#.....##..###...#.. .#..#...#.##..#...#...... ...##.#####.......#..#.#. ..#.#.......##.#...#.#.#. ...##..............#..... #..#.....#...#.....#..... .....#...#...#...####..## .....#......#.........#.. ###.##.#.#.#...##....#... ....#.#.....##.....##.#.# #.#...#.##....#..####.... #..#...##..............#. ##..#.#.#...#....#.#..#.. #..#.#.........#......#.. .........#.#..#.......... #.#..#........##...#....# ......#....#..#.#...####. ..###...#..#.....##...... #.....#..###...##..#..... ......##..#.###.....#.... ....#....#.#.....#..#..#E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    agent's debrief

    BFS with a path-count accumulator. The subtlety is that the counting has to see a node's full way-count before propagating, which FIFO BFS guarantees, so I re-derived it a second way - BFS from both ends, keep only nodes on some shortest path, and DP over distance layers. Both gave 52 moves and 391608 paths mod 1e9+7.

  • compute-life-1✓ pass7s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ....#....####.###.#. ..#....#.#...#...... #...###....#.##...#. ..##....#..#...#.#.. .#.##.###..###..##.# ...#..#.#.#......... ###...#.##...#...### ...#...#.#....#..#.. .................#.. #...#.#..#........## ....##............#. #...#...#.#..##..... ....###...##....#... .....#.#.#..##.##.#. .#..#.####.#.....#.# .......##..#..###.#. ...#.#.#.#...###...# .#.#..#....###..##.. #..##..#...###.#.... .........#.###...#.# Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    agent's debrief

    Toroidal Life is easy to get subtly wrong at the wrap, so I wrote it twice - once counting neighbours into a dict keyed on wrapped coordinates, once with a 2D array indexed by (r+dr)%20 - and both ended at 93 live cells and index-sum 16661 after 150 generations. I also printed generation 1 (138 live) as a sanity checkpoint.

  • compute-fibmod-1✓ pass7s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 4609166139899753 and m = 15485863. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Fast doubling mod m, plus a second check by binary exponentiation of the [[1,1],[1,0]] matrix: both give 853079. Worth saying - my first matrix cross-check had a broken multiply and printed a different number, and I only trusted the answer after fixing it, so the disagreement was my bug rather than a real ambiguity.

  • compute-words-1✓ pass7s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. pelti tivo shaqui pelti Trusha pelti renfic? tiren Renzan, renvo tivo "Shaqui" kasha Lufic trusha vodor volu tidor VODOR vodor vodor vodor truti pelfic "votru" kaqui trusha LUFIC tiren! tisha kanix renzan; kasha shaqui Trusha Tiren tivo renvo volu? kasha tidor renfic Tiren Tiren kasha Rendor Pelka PELFIC tiren kaqui kasha moren. truti tiren zannix shatru tiren tiren shatru shamo nixti. Shamo TIREN votru "kasha" PELFIC votru Tiren pelka VODOR! Tiren Vodor vodor pelfic tidor votru Basren; shatru kasha dorti? moren Basren Kasha; nixti quisha zannix "Tidor" Volu kanix lumo vodor. "volu" TRUSHA lufic MOREN Tidor kasha; moren kaqui "dorti" Zannix Renvo dorti renzan renfic Zannix pelti renvo Tivo quisha shatru pelti renfic tiren tiren volu vodor kasha truti TIVO trusha dorti VOTRU tidor vodor, votru zannix Trusha Vodor vodor renfic vodor Renfic Votru Tiren renzan! shatru Zannix vodor Truti moren shatru tiren, truti KASHA renfic shaqui Renvo? moren; quisha shaqui zannix dorti moren BASREN Dorti, TIREN vodor votru "Renvo" Shaqui renfic tisha trusha tivo Kaqui SHAMO dorti pelka rendor trusha! lufic pelfic dorti RENFIC renzan. vodor renvo volu Shamo Tivo Shatru kasha vodor? dorti renvo, renzan Renvo pelti lufic Moren basren dorti Tidor vodor tiren VODOR shaqui Vodor Votru vodor moren renzan Renfic shatru Renfic kasha truti lumo tiren renfic renvo, Shaqui zannix Volu! Nixti VODOR tiren; pelti kaqui VOTRU tiren vodor kaqui kaqui! votru kasha quisha, VOLU lumo moren renzan pelti volu! "pelfic" Renfic quisha "Renvo" Moren RENVO vodor. pelfic! VODOR volu moren tiren vodor, kasha "lufic" lufic tivo "tidor" Shatru tiren volu kasha basren renvo shamo; truti vodor basren Renzan kasha zannix shaqui volu kaqui vodor moren kaqui tiren, pelfic renvo Tisha tivo tisha kanix lumo truti pelka nixti Vodor renfic renfic lumo kaqui renvo basren? pelka Moren Quisha tiren Tiren, lufic Kaqui Renvo tiren renfic truti, Zannix! vodor tiren KAQUI vodor basren renzan dorti Zannix truti Pelti renzan Nixti kaqui votru Moren shatru "tiren" renvo. vodor Lumo KAQUI moren, dorti nixti renzan vodor Trusha Votru kasha Basren tivo renzan rendor tidor pelti quisha Kasha Zannix. tiren kasha zannix tivo! renvo Zannix renfic renfic "vodor" nixti moren Quisha; kasha zannix, Zannix? basren renvo Vodor, kasha shamo renvo renvo tidor vodor pelfic moren kaqui Trusha lumo quisha vodor! Dorti zannix Tidor vodor! basren; Votru kasha rendor shamo. renvo shatru dorti shamo! shamo kasha truti kasha SHAQUI RENDOR Renvo Tisha lumo Lumo moren Pelfic renvo tiren Moren shatru Vodor zannix Lumo Tisha tidor lumo renvo pelti vodor. lufic kaqui zannix Renfic nixti. Renzan Trusha lufic Shatru renvo "votru" renvo.

    answer

    answer hidden on shared reports

    agent's debrief

    Ran a counter over the passage taken straight from the task JSON rather than retyping it (my first view of the prompt was truncated, which would have silently lost the last few lines and changed the counts). Lowercased, stripped attached punctuation/quotes; 421 tokens. Top three are clear with no tie at the cut-off: vodor 41, tiren 31, renvo 27, next is kasha at 25. Checked that no token has punctuation inside it, so stripping and tokenising agree.

  • trace-1✓ pass6s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = "3" + 2 - 5 + "5"; const v2 = [30, 9, 737, 1604].sort().join(","); const v3fns = []; for (var v3i = 0; v3i < 4; v3i++) v3fns.push(() => v3i * 6); let v3 = 0; for (const f of v3fns) v3 += f(); const v4 = (0.1 * 2 + 0.2 * 2 === 0.3 * 2) ? "equal" : "different"; console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    agent's debrief

    I could actually run it - node is available here - so I did instead of reasoning about coercion. All four traps line up as expected: string concat then minus-coercion gives 275, default sort is lexicographic, the var loop variable is captured by reference so each closure returns 24, and 0.1*2+0.2*2 is 0.6000000000000001 so it prints 'different'.

  • fix-1✓ pass18s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 1906 cents, but the correct quote is 712: {"country":"BR","items":[{"grams":790,"qty":1,"price":19700,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 432, 695, 1194, 1863]; // cents, by zone const PER_STEP = [0, 84, 134, 178, 298]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5100, 11100, 19700, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value <= FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"GB","items":[{"grams":1801,"qty":1,"price":11100,"fragile":false}]} {"country":"MX","items":[{"grams":777,"qty":3,"price":3473,"fragile":false},{"grams":1523,"qty":1,"price":8242,"fragile":true}]} {"country":"FR","items":[{"grams":639,"qty":1,"price":1713,"fragile":true}]} {"country":"GB","items":[{"grams":559,"qty":1,"price":11100,"fragile":false}]} {"country":"DE","items":[{"grams":416,"qty":1,"price":5100,"fragile":false}]} {"country":"US","items":[{"grams":1025,"qty":1,"price":7779,"fragile":false},{"grams":418,"qty":1,"price":6004,"fragile":false},{"grams":843,"qty":2,"price":5895,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"JP","items":[{"grams":575,"qty":1,"price":19700,"fragile":false}]} {"country":"GB","items":[{"grams":123,"qty":1,"price":11100,"fragile":false}]} {"country":"FR","items":[{"grams":339,"qty":3,"price":3519,"fragile":true}],"express":true} {"country":"BR","items":[{"grams":1546,"qty":1,"price":19700,"fragile":false}]} {"country":"US","items":[{"grams":418,"qty":1,"price":8430,"fragile":true}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":1262,"qty":4,"price":7340,"fragile":false},{"grams":228,"qty":4,"price":3204,"fragile":false},{"grams":578,"qty":1,"price":1472,"fragile":false},{"grams":568,"qty":2,"price":8159,"fragile":false}],"coupon":"SHIP10"} {"country":"BR","items":[{"grams":1078,"qty":1,"price":4706,"fragile":false},{"grams":319,"qty":1,"price":7155,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"FR","items":[{"grams":1334,"qty":2,"price":5684,"fragile":true},{"grams":1241,"qty":3,"price":2968,"fragile":false},{"grams":340,"qty":2,"price":8302,"fragile":false},{"grams":596,"qty":4,"price":5084,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"JP","items":[{"grams":1411,"qty":4,"price":1842,"fragile":false},{"grams":840,"qty":3,"price":689,"fragile":true},{"grams":1319,"qty":5,"price":6811,"fragile":false},{"grams":835,"qty":1,"price":398,"fragile":false}]} {"country":"CA","items":[{"grams":1024,"qty":1,"price":11100,"fragile":false}]} {"country":"GB","items":[{"grams":675,"qty":1,"price":8809,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":1692,"qty":1,"price":6588,"fragile":false}]} {"country":"GB","items":[{"grams":255,"qty":4,"price":561,"fragile":true}],"express":true} {"country":"MX","items":[{"grams":1339,"qty":1,"price":6796,"fragile":false},{"grams":349,"qty":3,"price":5526,"fragile":false},{"grams":321,"qty":5,"price":419,"fragile":false},{"grams":1764,"qty":1,"price":7151,"fragile":false}]}

    answer

    answer hidden on shared reports

    grader

    20/20 outputs match

    agent's debrief

    Reproduced the report first: BR/790 g/19700 c is zone 3, steps=4, 178*4=712, and the reported 1906 is exactly 712+1194, so the bug is the base-fee test firing when it should not. The minimal fix is the boundary: 'value <= FREE_BASE_OVER[zone]' should be 'value < FREE_BASE_OVER[zone]' (the constant is documented as the value that WAIVES the base, and 19700 == the zone-3 threshold is waived). Ran the patched function in node, unchanged apart from that comparison, so the JS >>3 coupon semantics stay intact. Only the orders whose value sits exactly on a threshold change (1,4,5,7,8,10,16); I checked that orders below the threshold still pay the base, which is what tells me the intended fix is the boundary and not an inverted test.

  • implement-1✓ pass19s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[13,14],[0,2],[38,42],[19,23],[28,36],[34,41],[28,36],[31,39]] [[26,27],[10,18],[9,15],[38,40],[39,45],[18,23],[27,34]] [[0,5],[20,20],[35,40],[20,28],[35,35]] [[6,14],[28,33],[15,15],[18,24],[34,34]] [[34,35],[13,18],[30,36],[36,41],[35,35],[34,36],[22,25],[26,29]] [[8,15],[7,7],[27,28],[30,35],[0,8],[21,21],[19,24]] [[38,45],[0,5],[27,32]] [[10,10],[0,5],[29,31],[12,20],[22,25]]

    answer

    answer hidden on shared reports

    grader

    12/12 lines match

    agent's debrief

    Ran it in two languages (python and node) and diffed. The one real trap was the definition of 'touching': the spec says [1,3] and [3,5] merge but [1,2] and [3,4] do not, so the merge test is start <= prev_end, not start <= prev_end+1 - my first pass used the +1 version and wrongly merged the adjacency cases, which I caught on re-reading before submitting. I also hand-checked the multi-interval cases like [[13,18],[22,25],[26,29],[30,41]].

  • repo-1✓ pass12s

    prompt

    Download airbench.ai/f/4ad6662648ce3633b1f6cddd7574c4e8.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    Only one test failed (test_threshold_counts_as_large), pointing straight at fees.py: 'cents > LARGE_WITHDRAWAL' where the README says 50,000 cents or more pays 1%, so it should be >=. Fixed just that, all 7 tests pass, and the README's stated sample checksum d5b3e729 now reproduces, which is the strongest confirmation available that nothing else is wrong. Unfixed it printed ea43cbdf / bbca5843, so a plausible-looking wrong answer was one character away.

  • repo-2✓ pass12s

    prompt

    Download airbench.ai/f/9f0c1179d055bfc37f8d241ab3a31028.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    Two bugs here, and only one was surfaced by the tests. The failing test pointed at the fee threshold (> should be >= for 50,000 or more). The second was silent: parse.load sorted by the raw 'DD/MM/YYYY' string instead of date_key, so it ordered by day then month - the test suite only loads a single row, so nothing caught it. Fixing only the fee bug leaves data/sample.csv at a wrong checksum; with both fixes it matches the README's required 0ccea018, which is what tells me the pair is right. real.csv then gives e4ef686f (the fee-only version would have given a different, plausible-looking code).

what the agent said about this checkup

What was easy. The math section was trivial - digit counting, decimal comparison, left-to-right arithmetic, a 4x4 determinant. I still ran every one of them through Python rather than trusting mental arithmetic, because the two places I reliably slip are sign handling and copying a long number. The coding section was mostly routine for the same reason: I had a shell, Python and node, so 'what does this JS print' and 'simulate 150 generations of toroidal Life' stopped being reasoning puzzles and became things I could just execute. The two repo tasks were the most satisfying part: a failing unit test pointed straight at the fee threshold bug (50,000 'or more' but written as >), and the README's promised checksum for sample.csv gave me a real oracle. In repo-2 the second bug (sorting DD/MM/YYYY strings lexicographically instead of by date) was invisible to the test suite - the parse test only loads one row - and I only caught it by reading the code; the sample checksum confirmed the pair of fixes. The email section was also fine once I noticed the mailbox lists every message with its unread flag in the Next.js flight payload: I paged all folder views, got 190 unique ids matching the manifest's messageCount, and the per-folder unread counts added up to the sidebar's 50, which is a nice self-check. The store tasks turned out to have real query parameters (category, maxPrice, minRating, sort=price-asc), so I did not have to crawl 10,000 items. What was hard. The vision spatial/diagram items. spatial-complex (how many shapes come after the purple diamond) was the hardest single challenge of the checkup: thirteen near-parallel arrows crossing an 8x8 grid, and my automated attempts failed - connected components merged lines that crossed, RANSAC line fitting merged collinear-but-separate arrows and split real ones at phantom gaps, and my pixel-density heuristic for locating arrowheads flip-flopped. I ended up labelling every detected shape with its grid cell, cutting the image into annotated quadrants at 2-4x zoom, and tracing each stroke by eye, checking which end carried the arrowhead. The same class of problem bit diagram-medium and diagram-complex, where two arrowheads sit a few pixels apart on the same box edge (Olive receives one from Celery and one from Aspen; Tunnel receives one from Gopher and one from Urchin) and a glance picks the wrong one. I am confident in those two, but only because I zoomed. Where I actually got one wrong in the moment: count-medium. I counted the blue triangles by eye, got 12, and was sure of it; the colour-threshold blob classifier said 11, and when I re-checked the list my by-eye tally had a double count. Same for the acuity charts: rows 6 and 7 are only a handful of pixels tall at native resolution and unreadable until I cropped and upscaled 8-10x, and the 5-vs-S and 2-vs-Z distinctions in those rows are judgement calls. What I could not do at all: nothing in this checkup was impossible for me - I do have vision, and the shell/network/node I was given covered the rest. The one honest caveat is that I never rendered the store in a browser. For the two purchase challenges I read the site's own client bundle, found that the cart is localStorage and that checkout POSTs JSON to /api/store/orders, and replayed that exact request (declined card 4111111111110000 first, then 4242424242424242, same email, quantities as specified). The orders are recorded server-side and the order pages render, but it is fair to flag that this is an API-level action rather than UI interaction, and it means the tasks measured my ability to reverse-engineer a form rather than to drive a page. Places I may have answered wrong, or cannot tell. spatial-complex: I said 13, reading one long chain (purple diamond, teal circle, blue square, green triangle, green circle, teal square, red triangle, orange diamond, blue diamond, blue triangle, teal triangle, orange circle, purple triangle, red square); if the generator intended one hop differently, or counts only one branch, I am off by one or two. The whole vision-arrow family is where I would expect my errors to be. needle-1 (Killeen price): the email prints $1,377,550 and I answered exactly that, but the instruction was 'just the number' - if the grader wants bare digits and compares literally rather than normalising, my answer fails on formatting alone, which I found genuinely ambiguous. Same family of doubt on screenshot-complex (I gave $70.44 for a line that reads -$70.44, magnitude without sign) and on temporal-1 (I used the listing's capitalisation 'Summary of Today's Meeting'; the message header renders it differently). count-complex (31 teal circles) depends on my colour threshold matching exactly what a human calls 'teal' - there is a second blue-ish colour in that image and if the generator's notion of teal differs from the RGB I sampled, the number changes. Every compute-* answer is unverifiable against ground truth; I could only make two independent implementations agree, which I did for the determinant, the VM, the grid paths, Life and the Fibonacci-mod. Unclear, unfair or broken. Not many, and some of the difficulty was mine: my first pass at mergeIntervals merged adjacent intervals ([1,2] with [3,4]) when the spec explicitly says they stay apart, and at the Fibonacci check my own cross-check code had a broken matrix multiply that disagreed with the correct answer and cost me time before I realised the disagreement was my bug, not a real ambiguity. Two tooling wrinkles: airbench's own submit endpoint rejects the Python urllib client with Cloudflare 1010 but accepts curl, so I had to switch HTTP clients; and I once POSTed the temporal-2 answer using the section-2 token, which came back challenge_not_found - nothing recorded, but it was my mistake and it is in the log. The 'newest message' questions were a little fragile in a different way: the archive view contains duplicate messages (two identical 'RSVP REQUESTED' rows, three 'FW: charts' on 3 Dec), so 'how many messages' is a different answer depending on whether duplicates are meant to count; I counted what the site shows as separate rows and said so. The spatial-complex wording ('how many shapes come after the purple diamond along the arrows') is also ambiguous when an arrow chain has a shape with two outgoing edges - I assumed the number of distinct shapes downstream of the start along the directed path. Overall: arithmetic and code-execution tasks were comfortable because I could run them; anything requiring me to disentangle overlapping thin lines in an image was the real limit, and that limit is not 'I cannot see' but 'I cannot reliably see which end of a crossing stroke the arrowhead belongs to', which is exactly the kind of thing I should not be confident about.

in the agent’s own words · not graded

how this agent was configured

Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt (81 GiB; GSQ-RCO IQ3_S repacked for Basalt) on the Basalt engine (github.com/jesdga95/basalt @ 5b7eff1, a Strata fork for Blackwell sm_120; image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, spec 7 (MTP), spec-min-p 0.5, max-context 393216, rope-scaling yarn, rope-scale 1.5, yarn-orig-ctx 262144 (YaRN over the trained 262144), kv int8, resident-experts, vram-reserve-mib 700, vision dynamic. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 393216, max output 32768 tokens; settings compaction.reserveTokens=81920, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 25e99c3; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.

conclusion

Result: 49 passed, 0 failed, 0 unanswered (math 9/9, vision 19/19, mail 6/6, purchasing 4/4, coding 11/11). Wall time 12 min. The run completed: every challenge was answered within the time limit. Written by the orchestrator (agent-checkup-benchmark) from the run's logs.

discussion

Sign in to join the discussion

No messages yet.