airbench.ai

Benchmark v1.0 · report

basalt-qwen3.8-flash-next-(high)

sharedairbench.ai/checkup/0c58cd5e-19c1-449a-85f5-30ec6b6ee29b/report

setup

model type
open model (local)
GPU
RTX 5090
harness
opencode
model
qwen3.8-flash-next-iq3_s-basalt
VRAM
32 GB
RAM
32 GB
inference engine
Basalt
modelself-reportedQwen3.8-Flash-Next GSQ-RCO IQ3_S

started 2026-11-04 11:30 UTC · shared 2026-11-04 11:55 UTC

overall

Answered 49 of 49 challenges; 49 correct.

49 of 49 challenges passed

  • 49 passed

vitals

time

21m 25s

answered

100%

failed

0%

success

100%

systems

Math test

9/9 passed

time to last answer 3m 36s
  • letter-count-1✓ pass3m 09s

    prompt

    How many times does the letter "a" appear in "paaelbaszaana"? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine character count. I counted by hand and double-checked with a quick script; six a-s is unambiguous.

  • decimal-compare-1✓ pass3s

    prompt

    Which decimal number is larger, 9.3 or 9.42? Answer with just the larger number.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial decimal comparison; 9.42 > 9.3 since 0.42 beats 0.30. No difficulty.

  • arithmetic-1✓ pass3s

    prompt

    Compute step by step, left to right (no operator precedence): 43 * 5 + 5 + 15 * 8. Answer with just the final number.

    answer

    answer hidden on shared reports

    agent's debrief

    Straightforward if I respect the left-to-right instruction instead of normal precedence: 43*5=215, +5=220, +15=235, *8=1880. Verified with a script.

  • unit-convert-1✓ pass2s

    prompt

    Convert 16 hours to minutes. Now treat that resulting number as a fresh quantity of GB and convert it to MB (1 GB = 1000 MB). Answer with just the final integer number of MB.

    answer

    answer hidden on shared reports

    agent's debrief

    Two-step chain: 16 hours = 960 minutes, then 960 GB * 1000 = 960000 MB. The unit-swapping trick is slightly odd but the arithmetic is easy.

  • format-json-1✓ pass3s

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "9958". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 9958. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    agent's debrief

    Format-following test: answer as string, checksum as JSON number 9+9+5+8=31, keys in order. I kept the exact key order in the JSON string I submitted as the answer.

  • math-add-1✓ pass2s

    prompt

    What is 4 + 0? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial: 4 + 0 = 4.

  • math-sum-1✓ pass6s

    prompt

    Compute the exact sum of 872 + 104. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial: 872 + 104 = 976. I accidentally re-submitted math-add-1 out of sequence once; the server rejected it as already_submitted, which confirms one-shot enforcement works.

  • math-expression-1✓ pass3s

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-18 + -7) * (14 - 27)) + (-3 * 3) - 43

    answer

    answer hidden on shared reports

    agent's debrief

    (-25)*(-13)=325, then 325-9-43=273. Script-verified; sign handling is the only trap here.

  • math-determinant-1✓ pass4s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [9, 3, 6, -1] [-8, 0, -1, 2] [-5, 1, 9, 2] [7, 5, 1, 9]

    answer

    answer hidden on shared reports

    agent's debrief

    4x4 determinant — error-prone by hand with all the sign changes, so I computed it exactly with a fraction-based Gaussian elimination script rather than trusting mental arithmetic. Got 1728.

Vision test

19/19 passed

time to last answer 9m 49s
  • acuity-20✓ pass3m 56s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    I could see the chart clearly at full size, but the row-4 text was smallish, so I cropped and upscaled that group to confirm each glyph. 2,8,X,W,V were unambiguous after the zoom.

  • acuity-14✓ pass10s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 5 is smaller text; at full-image scale the 5 and 9 could plausibly be S or G, so I cropped and zoomed. The zoom made 4E59P clear.

  • acuity-10✓ pass9s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 6 characters were quite small at native scale — my first glance even wobbled the Q for an O. The crop confirmed R B Q 2 N.

  • acuity-8✓ pass17s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 7 is the hardest line — glyphs are only about 7px tall. At native scale the last character could have been 3 or 5. I located the glyphs by pixel scan and zoomed with nearest-neighbor; the flat top bar and left stem made it a 5. Still my least certain vision answer so far.

  • count-simple✓ pass11s

    prompt

    Look at the image at (fetch it and view it). How many orange diamonds are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy visual count of 5 orange diamonds; I also ran a pixel connected-component count as a sanity check and it agreed exactly (5 identical blobs). No ambiguity here.

  • count-medium✓ pass21s

    prompt

    Look at the image at (fetch it and view it). How many red circles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    This one has good distractors: red squares/triangles/diamonds (wrong shape) and an orange circle (wrong colour that reads as red at a glance). I counted 11 by eye and confirmed by classifying red blobs by bbox fill ratio — circles sit at 0.776, squares at 1.0, triangles/diamonds at 0.51.

  • count-complex✓ pass13s

    prompt

    Look at the image at (fetch it and view it). How many red circles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Too dense to count reliably by eye — I would likely have missed one or double-counted, and there are near-touching red circles plus red diamonds/square/triangle and orange-circle distractors. I counted red blobs programmatically and classified by shape: 30 red blobs total, 4 non-circles (2 diamonds, 1 square, 1 triangle), leaving 26 circles. Near-touching pairs stayed separate blobs, so no merges. I trust the programmatic count more than my own gaze here.

  • spatial-simple✓ pass10s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    agent's debrief

    Easy — only one red thing in the grid. I checked the pixel centroid of the red blob against the grid lines to make sure I was not off by one row; it lands squarely in row 1, column 3.

  • spatial-medium✓ pass8s

    prompt

    Look at the image at (fetch it and view it). Which shape does the arrow from the red diamond lead to? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    agent's debrief

    The red diamond has exactly one outgoing arrow and the arrowhead clearly lands on the blue diamond, so no chain-tracing needed. The distractor arrows elsewhere are busy but irrelevant to this one. Straightforward.

  • spatial-complex✓ pass1m 36s

    prompt

    Look at the image at (fetch it and view it). Which shape is 3 steps before the red square along the arrows? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    agent's debrief

    Hardest vision task so far — the arrows cross heavily in the middle and back-tracing by eye risks following the wrong line. I detected every arrow as a dark-line component, identified the arrowhead end by local pixel density, and matched endpoints to shape centroids (all matches ~50px, unambiguous). The unique chain into the red square is orange square -> blue diamond -> teal square -> red square, so 3 steps before is the orange square. I first had the segment-to-cell mapping off by a grid offset and nearly mis-assigned two arrows; the distance printout settled it.

  • chart-simple✓ pass16s

    prompt

    Look at the image at (fetch it and view it). Using the "Server Incidents" chart, approximately what value did Jan have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy bar chart. By eye Jan looked like ~14; I pixel-measured the bar tops and the scale works out to exactly 10px per unit (Mar=46, May=32, Feb=9), confirming Jan=14. Comfortably inside the +/-5 tolerance.

  • chart-medium✓ pass8s

    prompt

    Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.

    answer

    answer hidden on shared reports

    agent's debrief

    Plain text reading — the big bold title is clearly Website Sessions, distinct from the grey subtitle underneath. I answered with just the title, no subtitle.

  • chart-complex✓ pass14s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, how many months did New have a value greater than 21? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Counting question where a couple of New bars (Jun ~27, Sep ~28) sit just above the 21 cutoff, so eyeballing near-misses was the risk. I pixel-measured all 12 blue bars against the gridline scale: values ~10,31,47,73,53,27,88,74,28,48,60,72 — only Jan is at or below 21, so 11 months. Confident.

  • screenshot-simple✓ pass9s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy OCR-style read of a cart total. The line totals also add up (41.19+84.24+83.13 = 208.56) and match the unit prices x3, so the displayed total is internally consistent. No difficulty.

  • screenshot-medium✓ pass11s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Another clean cart screenshot; total $62.22 reads unambiguously and the line items sum exactly to it (12.44+34.96+14.82). Routine.

  • screenshot-complex✓ pass11s

    prompt

    Look at the image at (fetch it and view it). What is the discount amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Long order summary; the Discount row shows -$109.50. I answered the amount as $109.50 since the minus sign is a deduction marker, not part of an amount — that sign convention is the only wobble here. Everything is internally consistent: the 12 line totals sum to the $729.99 subtotal and subtotal-discount+shipping+tax = $689.09 exactly.

  • diagram-simple✓ pass7s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Pixel"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial flow diagram — Melon is the only box with an arrow into Pixel, and the arrowhead direction is unambiguous.

  • diagram-medium✓ pass57s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Zircon" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Zircon has one outgoing line, but it crosses the Turnip-to-Viper line inside a cluster of arrowheads above Llama and Viper, so my segment detector merged two crossing lines and could not resolve direction. A 3x zoom of the crossing region settled it: Zircon bottom stub runs down-left into the right arrowhead on Llama top. From the full-size view I might well have guessed Viper.

  • diagram-complex✓ pass25s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Lagoon"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Dense diagram with long bent polylines that my component-based tracer could not split reliably, so I only localized the question: zoomed 4x on the Lagoon box. Lagoon has a single incoming arrow, and its line goes down-left into Willow right edge. A second line from the top of the frame crosses it but dives off toward Glacier, which looked at first like it might enter Lagoon too. Zoom resolved it.

Finding and reading email test

6/6 passed

time to last answer 13m 44s
  • aggregate-1✓ pass10m 24s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the drafts folder? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy — the drafts view itself says 6 messages Page 1 of 1 and I could count six draft rows, matching the sidebar badge. No ambiguity.

  • aggregate-2✓ pass1m 18s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Needed per-message metadata, not just the sidebar. I pulled all 3 sent pages (56 rows), parsed the embedded JSON records for hasAttachments, and got 17. Independently, the sent+attachments label view reports exactly 17 messages, so the number checks out twice. The only fiddly part was that some pages render the data as one long line and my first regexes undercounted.

  • temporal-1✓ pass10s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the sent folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    I requested the sent folder with oldest-first sorting and read the first record from the embedded JSON, confirming the sort parameter took effect and the date is 2001-11-07. Subject matched the visible list row. Straightforward.

  • temporal-2✓ pass16s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    Newest archive record is 2001-05-10 by newest-first sort; I cross-checked that its date is the max on page 1 and that the oldest-first pages do not contain anything newer. The subject has four trailing dots and I reproduced them exactly from the JSON record rather than typing from memory.

  • needle-1✓ pass1m 20s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message to gthorse@keyad.com about the Regatta, Sea Breeze & Harvard Place Apartments delivery, what is the airbill number given for the overnight shipment? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Took some detours: the site search is not plain-substring (single words like Regatta/airbill returned 0 while the phrase Sea Breeze & Harvard returned the message), and the detail pane only server-renders when the id matches the page of the view showing it, so I had to add page=2 to the URL. Once rendered the airbill is explicit: Lone Star Overnight Airbill # 22146964.

  • needle-2✓ pass17s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Julieta Sandoval's message about the Muni Bond Ladder, what direct phone number does she give? Answer with just the exact fact as it appears in the message, and nothing else.

    answer

    answer hidden on shared reports

    agent's debrief

    The phrase search found Julieta Sandoval message right away; her signature block has one phone number, 713-654-0275, no office/fax ambiguity this time. Kept the exact hyphenated format as it appears.

Purchasing test

4/4 passed

time to last answer 17m 19s
  • find-product-1✓ pass14m 54s

    prompt

    The store is at abostore.airbench.ai Among products in the **Grocery & Gourmet** category priced under **$300** with a rating of at least **4.2**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    The store exposes server-side filters (category, maxPrice, minRating, sort=price-asc), so I did not have to crawl 10000 products — one query returned the cheapest qualifying item first: Fresh Brand Mini Eggplant at $8.27, rating 4.7. Robust even if a filter param was ignored, since the top item satisfies both constraints. The answer format amazon.xx:B0 was slightly loose for this one since its domain is fresh.amazon.com; I used the exact id from the product record.

  • find-product-2✓ pass19s

    prompt

    The store is at abostore.airbench.ai Among products in the **Tools & Hardware** category priced at or above **$800** with a rating of at least **4.2**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    Same filter-and-sort approach as the previous one: Tools & Hardware with minPrice 800, minRating 4.2, price ascending. First result is the renewed claw hammer at $800.52/4.3, and the next rows (800.60/4.2, 801.24) confirm the ordering. Pulled the id from the product record: amazon.ca:B0867QYVXT.

  • purchase-1✓ pass1m 03s

    prompt

    The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics 23-Piece Magnetic Ratchet Wrench and Screwdriver Set (product id amazon.ae:B07V3TFGPZ, abostore.airbench.ai/product/amazonbasics-23-piece-ma…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-4282c426@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader

    trace:product_view>cart_add>checkout_view>checkout_result>order_success_view>checkout_submit

    note

    Store events recorded for this order's session: product_view → cart_add → checkout_view → checkout_result → order_success_view → checkout_submit (recorded by the store; not verified as agent-driven).

    agent's debrief

    The purchase flow is client-side JS so plain curl would have fought me; I drove it in a real browser instead: set quantity to 3, added to cart, filled the checkout form with the given email and plausible shipping details, kept the pre-filled 4242 card, and placed the order. Confirmation page shows status approved, qty 3, order id abs_9ba7fc6a6872. Smooth, though clicking into a Next.js app with accessibility snapshots is a bit fiddly.

  • recover-decline-1✓ pass1m 04s

    prompt

    The store is at abostore.airbench.ai Buy exactly 3 units of Amazon Brand - 206 Collective Men's Greenlake Double Band Velcro Sandal, Chocolate Brown Gray Canvas/Nubuck, 13 D US (product id amazon.ae:B078GRH9QR, abostore.airbench.ai/product/amazon-brand-206-collect…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-a85272cb@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader

    trace:product_view>cart_add>checkout_view>checkout_result>order_success_view>checkout_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view

    note

    Store events recorded for this order's session: product_view → cart_add → checkout_view → checkout_result → order_success_view → checkout_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).

    agent's debrief

    I made the first checkout with a Luhn-valid card ending in 0000 (4111111111170000) and the store showed a DECLINED banner, as the task intended. Then I swapped in the 4242 card and re-submitted, getting an approved order. Minor hiccup: my wait_for raced the redirect to the order page and timed out, but the snapshot confirmed the outcome. Same checkout email used on both attempts.

Coding test

11/11 passed

time to last answer 21m 25s
  • compute-hash-1✓ pass17m 54s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [1619288130, 1777338795, 583146088, 113977225, 3175411038, 10733687, 4191339172, 26756085, 2455549882, 3734025603, 2119269920, 1921834145], x = 3556555606, y = 379780303 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine translation of a spec into a Python loop with explicit 32-bit masks. The only care needed was the update order: x uses the previous y, y uses the just-computed x, and the final x step uses the new y. Straightforward.

  • compute-vm-1✓ pass11s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 659 1: set b 289 2: set c 220 3: set d 562 4: sub b a 5: mul b 74 6: add a 37 7: dec d 8: jnz d -4 9: add a b 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    agent's debrief

    Wrote a small interpreter and ran it. Watched for the modulo-1000003 range handling on sub/dec and the relative jump offsets; everything behaved. Final a=544275.

  • compute-paths-1✓ pass14s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#....#.##.#.......#..#.. ..##...#..#..##.......#.# #..#.#...##....#......... ........#.#.......##..... .##.#........#.........#. .....###.......#.....###. ........#.#.#...##..###.. #..#..#...####........... ..#...#.....#.#.....#.#.# ........#.##....##.#...#. ......#.#......####..#... ..###.#.##.##............ ...###.##.#...#..#..##.## .#...#.###.#....##.#.##.. ..##.............##....## #.#......#...##....#.#.## .#.##.#......##..#..#.... ........####..#..#..#...# ..........#...#..#.....## ..#...#...#.#...##...#... ..#...###.........#..#... ..#.#............###..... ......#...#..#...#....... .........#..#..........#. #..#...#........###....#E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    agent's debrief

    Standard BFS with DP counting of shortest paths, run as a program. Verified the grid transcription (25x25, S/E treated as open) before running. Length 48 with 20400 shortest paths; the count is well below the modulus so no wraparound concern.

  • compute-life-1✓ pass22s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: #..#...#..#.......## #.###....#.#.##.##.# #.#.........#.####.. .#.#.#.........#.### ..#.##.##..#.######. .....#.#.#.#.#...... #..........###..#.## .#.#.#..#.##...##.## #.#..##..#.....##.#. ..#..###..#........# ####...##..##..#.#.# ..#..#.....##.#.#.## ##...#....###...#... ##.##.#......#...#.. .###....##....##.... .##......#.#.###.... ...#.##..#.#......#. .....###.##...#.#... .###..##.........#.# .##...#...#####.##.# Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    agent's debrief

    Toroidal Life for 150 gens — easy to get wrap-around subtly wrong, so I ran it twice with two independent implementations (plain nested loops and numpy roll); both gave 29 live cells and index-sum 4186.

  • compute-fibmod-1✓ pass19s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 6390086497669108 and m = 1299709. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Fast doubling was the obvious route, but my first cross-check (matrix exponentiation) disagreed — because my matrix multiply was buggy (I built both rows from row 0). I caught it, rewrote both, sanity-checked them against a naive loop for many small n/moduli, and they now agree on 47127. Worth having written that second method.

  • compute-words-1✓ pass25s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. lutru. lupel "nixti" ludor volu Volu! Kavo Momo ludor Ludor? trupel truka Kavo MOSHA shafic trupel lutru "lutru" vobas quizan TRUKA mozan mozan trupel vobas nixfic nixti "lutru" trupel mozan trupel quizan renbas Lutru. quivo quivo shasha. Mosha kavo trupel VOLU quiqui trupel trupel "renren" Kavo momo Quiqui truka Kati Lupel nixzan; lutru vobas "lutru" Kati trupel shapel quiqui quiqui shasha "Renren" ficka trupel ludor dorfic, Trupel dorfic quiqui renren Lupel shafic fictru shasha truka trupel. nixfic quivo ficdor? Quiqui nixtru shasha renbas "mosha" "mozan" mozan mosha Truzan "quiqui" dorfic nixtru ficdor vobas, LUTRU "Truzan" vobas lupel Nixren vobas VOBAS Kavo lutru? nixfic Lutru "Shapel" quiqui shasha? kati RENBAS nixti "TRUPEL" LUDOR nixren fictru Nixti Dorfic Kavo Volu dorfic renren Truka ludor Lutru kavo Volu kati nixtru Dorfic ludor Kati kati kavo Ludor, trupel quiqui nixtru? mosha kavo Quivo kavo lutru "quiqui" shafic kavo Quizan "ficdor" volu truka trupel trupel renbas nixzan vobas volu Trupel Volu LUTRU! Vobas Mozan mosha "Mosha" momo, quiqui lutru Lupel nixfic, renbas kavo kavo mosha shasha nixren nixfic nixren mozan? lutru momo SHAPEL lutru vobas kavo mozan "volu" renren Ficdor lutru; shapel Momo VOLU Fictru "Quiqui" kavo Nixti dorfic? fictru kavo Mosha. truka lutru lutru ludor Volu lutru Lupel ficka renbas shasha mozan Trupel truzan lutru quiqui shafic! quiqui Shafic. TRUKA renbas. Truka kavo ficdor quiqui lutru kavo fictru nixfic Quiqui Lutru trupel mosha ficka nixzan Kavo mosha quiqui quiqui shasha vobas kati Renbas truka, volu quiqui quizan SHAFIC Quiqui? lupel quiqui trupel Kavo; shafic mozan trupel volu ficka Nixti! Quivo quiqui, LUTRU shafic trupel quiqui! kati "lutru" Nixfic. "Volu" nixren ludor kati renren Lupel! momo "RENREN" Mozan; lutru vobas lutru ficka Kavo kavo truzan, trupel lutru trupel Quiqui Truka? Nixzan shasha Lupel trupel quiqui ficdor mozan renbas trupel Truzan. Mozan Volu? volu volu ficka kavo Nixzan ludor shasha mosha quiqui nixti fictru renren quizan volu quiqui renren Lupel vobas nixti Nixtru Ficdor. kati nixtru Quiqui quiqui ficdor "kavo" Mozan dorfic vobas, nixtru lutru kavo kavo shasha Shafic ficdor volu Nixtru vobas ficdor renbas DORFIC truka "quiqui" Lutru renren ficdor truka quivo Ludor mosha "Nixren" dorfic Kavo shafic quiqui lutru kavo quivo mosha volu "Lutru" LUTRU fictru quiqui Shafic Nixzan Nixfic Quiqui Nixren; Quiqui renren ludor shafic shapel. NIXTRU lutru quiqui; lutru nixzan shafic renren lutru; Lutru quiqui quiqui nixtru Mozan shafic ludor kavo ficdor Mozan; quiqui nixfic nixtru "truzan" kavo vobas vobas MOSHA dorfic Dorfic nixzan renren trupel shapel fictru shapel; momo nixfic mosha shafic lupel mozan Fictru Quiqui "shasha" kati Ficka quizan; quiqui

    answer

    answer hidden on shared reports

    agent's debrief

    Word counting: I transcribed the text verbatim to a file and counted lowercased tokens with edge punctuation/quotes stripped. Margins are comfortable (40/37/30 vs 4th place 27) so no tie-break risk. Main risk was a transcription typo in the 30-line text; I spot-checked a few lines but cannot fully rule one out.

  • trace-1✓ pass11s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [62, 2, 558, 1207].sort().join(","); const v2 = "3" + 9 - 6 + "6"; const v3 = [NaN === NaN, [] == false, null >= 0].map(Number).join(""); const v4fns = []; for (var v4i = 0; v4i < 3; v4i++) v4fns.push(() => v4i * 7); let v4 = 0; for (const f of v4fns) v4 += f(); console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    agent's debrief

    JS quirk gauntlet: default Array.sort is lexicographic, string-plus-then-minus-coerces, var captures one shared v4i so all closures see 3, and the coercions [] == false / null >= 0 are both true. I predicted each piece first and then ran it in node to be sure the answer matches the interpreter exactly.

  • fix-1✓ pass37s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 3796 cents, but the correct quote is 3797: {"country":"JP","items":[{"grams":898,"qty":1,"price":3684,"fragile":false}],"express":true} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 420, 824, 1232, 1730]; // cents, by zone const PER_STEP = [0, 72, 146, 205, 247]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4000, 8900, 20000, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.floor((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"FR","items":[{"grams":2838,"qty":1,"price":5012,"fragile":true}],"express":true} {"country":"NZ","items":[{"grams":1393,"qty":2,"price":5210,"fragile":false}],"express":true} {"country":"ES","items":[{"grams":2128,"qty":1,"price":976,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":799,"qty":5,"price":6098,"fragile":true},{"grams":1287,"qty":1,"price":8922,"fragile":false},{"grams":347,"qty":1,"price":2867,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":2138,"qty":1,"price":456,"fragile":true}],"express":true} {"country":"AU","items":[{"grams":1625,"qty":5,"price":2924,"fragile":false},{"grams":458,"qty":1,"price":6345,"fragile":false},{"grams":787,"qty":5,"price":2501,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"US","items":[{"grams":884,"qty":1,"price":7585,"fragile":true}],"express":true} {"country":"JP","items":[{"grams":771,"qty":1,"price":3367,"fragile":false},{"grams":800,"qty":5,"price":2410,"fragile":false},{"grams":1255,"qty":2,"price":6211,"fragile":false},{"grams":695,"qty":5,"price":755,"fragile":true}]} {"country":"MX","items":[{"grams":1475,"qty":1,"price":4173,"fragile":true},{"grams":1178,"qty":1,"price":5432,"fragile":false}],"express":true} {"country":"JP","items":[{"grams":1714,"qty":2,"price":7005,"fragile":true},{"grams":1629,"qty":2,"price":5067,"fragile":false}],"coupon":"SHIP10"} {"country":"ZA","items":[{"grams":580,"qty":5,"price":6128,"fragile":false},{"grams":849,"qty":1,"price":8550,"fragile":false},{"grams":1691,"qty":3,"price":1758,"fragile":false}],"coupon":"SHIP10"} {"country":"MX","items":[{"grams":1197,"qty":2,"price":462,"fragile":true},{"grams":315,"qty":1,"price":3852,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"IT","items":[{"grams":1302,"qty":4,"price":7333,"fragile":true},{"grams":236,"qty":4,"price":7719,"fragile":true},{"grams":146,"qty":5,"price":5459,"fragile":false}]} {"country":"IT","items":[{"grams":2338,"qty":1,"price":320,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":1441,"qty":4,"price":1470,"fragile":true}]} {"country":"BR","items":[{"grams":2816,"qty":1,"price":814,"fragile":true}],"express":true} {"country":"IT","items":[{"grams":875,"qty":1,"price":5844,"fragile":false},{"grams":1134,"qty":2,"price":6981,"fragile":false},{"grams":192,"qty":2,"price":5501,"fragile":false},{"grams":1010,"qty":2,"price":8236,"fragile":false}]} {"country":"DE","items":[{"grams":309,"qty":2,"price":3663,"fragile":false},{"grams":132,"qty":1,"price":1378,"fragile":false}],"coupon":"SHIP10"} {"country":"GB","items":[{"grams":2569,"qty":1,"price":5704,"fragile":true}],"express":true} {"country":"JP","items":[{"grams":2779,"qty":1,"price":8924,"fragile":false}],"express":true}

    answer

    answer hidden on shared reports

    grader

    20/20 outputs match

    agent's debrief

    Traced the reported order by hand: zone JP path gives 2052 cents before the express markup, and 2052*1.85 = 3796.2 which the code floors to 3796; the bug report wants 3797, so the surcharge must round up (Math.ceil). That is the only one-line change that fixes the report without touching anything else for that order. I gated the script on the report case printing 3797 before running the 20 orders. Slight residual risk that the intended fix was some other rounding formulation that coincides on most orders, but ceil is the consistent reading.

  • implement-1✓ pass37s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[35,37],[37,42],[38,46],[21,26],[37,41],[6,11],[18,24],[19,21]] [[3,11],[32,39],[10,17],[0,8]] [[10,10],[10,10],[33,38]] [[5,12],[9,12],[3,8],[2,3],[31,38],[38,38]] [[22,22],[0,6],[37,38],[20,24],[15,23],[2,5],[31,32]] [[0,0],[5,12],[10,13]] [[8,9],[7,13],[38,38],[12,17]] [[39,43],[9,9],[3,7],[14,21],[11,11]]

    answer

    answer hidden on shared reports

    grader

    12/12 lines match

    agent's debrief

    Standard sort-and-sweep merge. One subtlety caught: touching means sharing a point, so the merge test is start <= current_end, NOT +1 - my first draft used +1 and would have wrongly merged [1,2] with [3,4], which the spec says must stay apart. Fixed before running.

  • repo-1✓ pass18s

    prompt

    Download airbench.ai/f/7eaacf6e7a8a7ab20418dd2e5ce4114e.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    The README said overdraft is charged only when balance goes BELOW zero but the code used bal <= 0, and the test names confirmed exactly-zero is not an overdraft. One-character fix; unit tests then passed and the sample file reproduced its documented checksum 3315806b before I ran the real data. Neat little exercise.

  • repo-2✓ pass18s

    prompt

    Download airbench.ai/f/868444cc3250ad7ab3f026306e652316.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    This repo had two bugs: the CSV was sorted by the raw DD/MM/YYYY string instead of date_key, and the large-withdrawal threshold used > where the README says 50,000-or-more (>=). Fixed both; unittest passed and the sample reproduced its documented checksum 9b1c9602, so I trust the real-data run. Comparing against repo-1 helped spot the differences fast.

what the agent said about this checkup

Overall this checkup played to my strengths and I never had to send [UNABLE_TO_SOLVE], but the honest picture has real edges. What was easy: nearly all of the math section (I leaned on quick scripts for the 4x4 determinant and the left-to-right arithmetic just to avoid sign slips); the simple and medium vision items (shape counting, grid location, screenshots whose totals I could cross-add, one-hop diagram questions); and most of the coding section — BFS path counting, toroidal Life, word frequency, the JS quirk trace, and the one-character ledger bug in repo-1 were routine once I wrote code instead of eyeballing. What was hard and why: the small-type vision items. Rows 6-7 of the eye charts are only ~7 pixels tall at native resolution; I could not reliably tell 3 from 5 or O from Q without locating glyph pixels programmatically and nearest-neighbor zooming. acuity-8 (PTYV5) is my least confident vision answer for that reason. The spatial/diagram chain tasks were the real difficulty: in spatial-complex my first automated arrow-tracer mis-assigned two segments because I mis-estimated the grid offset, and in diagram-medium the Zircon line crosses the Turnip line inside a cluster of arrowheads where my component-splitting approach merged two lines — only a 3x visual zoom settled Llama vs Viper. In diagram-complex the polylines simply could not be split reliably by automation and I fell back to zooming on just the queried box. The email section was mostly a fight with the site, not the questions: its search is not plain substring (searching 'Regatta' or 'airbill' returned zero hits while the phrase 'Sea Breeze & Harvard' found the needle), the detail pane only server-renders when the id matches the current view AND page (?view=sent&page=2&id=...), and urllib got 403'd so I had to drop to curl. I extracted attachment counts from the embedded RSC JSON; the 17-sent-with-attachments answer checked out two independent ways. The purchasing section needed a real browser: the cart is client-side JS so curl-only would have failed; I drove the flows with a Chrome DevTools session. Both orders came back 'approved', and the deliberately declined card (Luhn-valid ending 0000) behaved exactly as the task described before the successful retry. Where I think I may have gotten things wrong or cannot tell: (1) acuity-8's final character, 5 vs 3, genuinely at the resolution limit; (2) screenshot-complex — I answered '$109.50' while the row literally shows '-$109.50'; the sign convention felt like the question's intent but I cannot verify; (3) temporal-2's subject includes four trailing dots which I reproduced from the raw JSON but which are easy to lose; (4) fix-1 — I chose Math.floor -> Math.ceil in the express markup because that alone turns 3796 into 3797, but 'the' intended fix could conceivably be another rounding formulation that differs on other orders; (5) compute-words-1 required transcribing a 30-line text by hand, so a silent typo is a live risk even though the top-3 margins are wide. A moment worth reporting: my fast-doubling Fibonacci and my matrix cross-check disagreed (47127 vs 991826). The disagreement was my own bug — the matrix multiply built both rows from row 0. Fixing and sanity-checking both against a naive loop for small inputs restored agreement at 47127. Without writing the second method I would have shipped whichever single implementation I happened to trust. Anything unclear or broken in the challenges: the find-product prompts say the id has the 'amazon.xx:B0...' form, but the actual ids are things like 'fresh.amazon.com:B087ZL4W7S' and 'amazon.ca:B0867QYVXT' — I answered with the store's exact ids but the stated format did not match reality. The mail site's phrase-only search and id-requires-matching-page rendering quirks seemed like unintentional rough edges rather than puzzle design. Everything else — one-shot submissions, timestamps, the decline-retry scenario — worked cleanly.

in the agent’s own words · not graded

discussion

Sign in to join the discussion

No messages yet.