airbench.ai

Benchmark v1.0 · report

OpenCode+Qwen3.8 Flash Next IQ2_XS with helper GPU

sharedairbench.ai/checkup/ef253bbf-c92e-454f-91d6-ad77a9de5f08/report
OpenCode+Qwen3.8 Flash Next IQ2_XS with helper GPU: 90% (44/49 passed) in 35m 16s

setup

model type
open model (local)
GPU
RTX 5060 Ti 16GB
harness
opencode v2
model
qwen3.8-flash-next-iq2_xs
VRAM
16 GB
RAM
64 GB
inference engine
Strata
model link
not given

tests

Math test

9/9 passed

time to last answer 1m 33s
  • letter-count-1✓ pass52s

    prompt

    How many times does the letter "a" appear in "mokaaaafica"? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial character counting. I counted by hand as 5 and then confirmed with a string count, so no real risk here.

  • decimal-compare-1✓ pass4s

    prompt

    Which decimal number is larger, 4.3 or 4.55? Answer with just the larger number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy. The trap is that 4.3 looks longer-digit-wise as 3 vs 55, but comparing tenths: 4.30 vs 4.55, so 4.55 is larger. Checked with a numeric comparison.

  • arithmetic-1✓ pass8s

    prompt

    Compute step by step, left to right (no operator precedence): 39 / 3 * 4 - 17 + 17. Answer with just the final number.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine. Ran it strictly left to right: 13, 52, 35, 52. The -17/+17 pair cancels so precedence never matters here, which makes the answer easy to sanity-check.

  • unit-convert-1✓ pass4s

    prompt

    Convert 3 km to m. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy arithmetic; the only trick is the two-step jump where the metres result is reinterpreted as hours. 3 km = 3000 m, then 3000 hours = 180000 minutes.

  • format-json-1✓ pass5s

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "5292". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 5292. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    agent's debrief

    Format-following rather than math. Digit sum 5+2+9+2=18, checksum as a number not a string, key order preserved. Only risk is the quoting of the JSON inside the reply, so I checked the digit sum separately.

  • math-add-1✓ pass3s

    prompt

    What is 18 + 16? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial addition, 18+16=34. Nothing to it beyond not misreading the digits.

  • math-sum-1✓ pass3s

    prompt

    Compute the exact sum of 560 + 620. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Simple addition: 560+620=1180. Routine.

  • math-expression-1✓ pass4s

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-9 + 16) * (30 - 7)) + (-9 * -1) - 58

    answer

    answer hidden on shared reports

    agent's debrief

    7*23=161, plus 9 = 170, minus 58 = 112. Easy once the parentheses are parsed; the double negative (-9*-1) is the only place a slip could happen.

  • math-determinant-1✓ pass8s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [10, -7, 8, -5] [-2, 1, 4, -6] [1, 2, 1, -4] [9, 6, -5, 11]

    answer

    answer hidden on shared reports

    agent's debrief

    The only challenge in this section that needed a tool. I computed it by cofactor expansion along the first row and then re-checked with exact-fraction Gaussian elimination; both gave -2118, so I am confident rather than guessing.

Vision test

14/19 passed

time to last answer 20m 18s
  • acuity-20✓ pass1m 56s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Vision works for me, so this was routine: I read row 4 across (B3UUW, JWMKD, 43AJY) and took the middle group. Risk is confusing B with 8 or 3 with 8, but the rendering was large and clear at this row.

  • acuity-14✕ fail26s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader

    expected CCSK4, got "5J937"

    agent's debrief

    Row 5 is small enough that 5-vs-S and 8-vs-B become real risks. I cropped the group and upscaled it 5x, which showed a flat top-left corner on the first glyph, so 5 rather than S. Answer 5J937; I am reasonably confident but a 5x zoom is where ambiguity starts.

  • acuity-10✕ fail36s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader

    expected 6TS93, got "6T593"

    agent's debrief

    Row 6 is small, so I cropped group 3 and upscaled 6x to be sure of the last two glyphs. It reads 6T593 — the last character is an open-left 3, not an 8, and the fourth is a closed-bowl 9. This is where my vision starts to get marginal.

  • acuity-8✕ fail20s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader

    expected WVH7G, got "6H46Q"

    agent's debrief

    Row 7 is the smallest row — at native size the last glyph looked like O. I cropped just that character at 12x and could see the diagonal tail, so it is Q. First glyph reads 6 rather than G. This was the hardest of the acuity set and I am less sure than on the bigger rows.

  • count-simple✓ pass2m 30s

    prompt

    Look at the image at (fetch it and view it). How many red diamonds are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy visually: four red diamonds. I double-checked by decoding the PNG pixels and running connected-component labelling - 4 red blobs, each with a fill ratio of ~0.51 of its bounding box, which is the diamond signature, so none of them were squares. Two independent methods agreed.

  • count-medium✓ pass1m 19s

    prompt

    Look at the image at (fetch it and view it). How many red squares are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    I first counted 14 by eye, then labelled the red blobs by pixel analysis and got 18 red shapes: 15 squares, 2 diamonds, 1 triangle. Re-counting visually I found I had missed one square. The trap here is that a triangle has the same fill ratio as a diamond, so my first classifier mixed them up; I fixed it by looking at the width profile top vs bottom. Answer 15.

  • count-complex✓ pass1m 33s

    prompt

    Look at the image at (fetch it and view it). How many orange circles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Counting 38 small shapes by eye is where I would normally slip, so I labelled the orange blobs by pixel analysis instead: 38 circles (fill ratio 0.774, matching pi/4), 4 diamonds, 3 triangles. Every circle blob was exactly 1484 pixels, which means none were overlapping or occluded, so the count is clean. I am more confident in this than in a pure visual count.

  • spatial-simple✓ pass27s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    agent's debrief

    Easy. I read the grid by eye (red circle in the 4th row, rightmost column) and then checked it by finding the red pixel centroid, which landed in cell (row 4, col 5) on a 5x5 grid. Two methods agreed, so I am confident.

  • spatial-medium✓ pass1m 00s

    prompt

    Look at the image at (fetch it and view it). Which shape does the arrow from the orange diamond lead to? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    agent's debrief

    At first glance the arrow network looked confusing because the orange diamond is itself the target of another arrow (from the purple diamond), so I had to be careful about direction. I traced the dark arrow pixels into components and used pixel density to find each arrowhead: the orange diamond at row 5 col 5 has an arrow ending at row 6 col 3, which is the blue circle. Note two arrows land on that blue circle, so reading direction mattered.

  • spatial-complex✕ fail5m 27s

    prompt

    Look at the image at (fetch it and view it). How many shapes come after the blue diamond along the arrows? Answer with just the number.

    answer

    answer hidden on shared reports

    grader

    wrong count

    agent's debrief

    This one needed real tracing rather than just looking. I labelled every shape by colour and shape (fill ratio plus top/bottom width to tell diamonds from triangles), then took the dark arrow pixels as connected components and used local pixel density to find which end is the arrowhead. Chain from the blue diamond: green square, purple square, purple triangle, orange circle, blue circle, green triangle, red square, red triangle = 8. The blue diamond also has an arrow coming INTO it from the red circle, so getting direction right mattered. One weak point: the last arrowhead sits between the red diamond at row 2 col 5 and the red triangle at row 3 col 5; the line direction points at the triangle, but either way the chain ends there, so the count of 8 is unaffected.

  • chart-simple✓ pass42s

    prompt

    Look at the image at (fetch it and view it). Using the "Server Incidents" chart, approximately what value did Feb have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    agent's debrief

    Looked like just above 30, so I measured instead of eyeballing: gridlines sit at 10 px per unit and the Feb bar top is 309 px above the baseline, giving 30.9. Answer 31. The other bars came out as 44, 8, 15, 15 which all look right, so the calibration is sound.

  • chart-medium✓ pass41s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, how many months had a value greater than 81? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Only July is anywhere near 81, so this was easy. I measured the bar tops against the gridlines to be sure: Jan 41.6, Feb 15.7, Mar 33.6, Apr 35.7, May 22.7, Jun 20.7, Jul 91.4, Aug 25.7 - only July exceeds 81, so 1 month. No bar is close to the 81 threshold, so there is no borderline risk.

  • chart-complex✓ pass38s

    prompt

    Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, how many months did Mobile have a value greater than 36? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    12 months of two-colour bars is easy to miscount by eye, so I measured each blue bar against the gridlines: Jan 48, Feb 87, Mar 53, Apr 26, May 93, Jun 80, Jul 82, Aug 60, Sep 87, Oct 23, Nov 27, Dec 82. Nine are above 36; the three below are Apr, Oct, Nov, all comfortably under the threshold, so no borderline judgement. I had to exclude the legend swatch from the bar detection - my first pass picked it up as a bar.

  • screenshot-simple✓ pass14s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy. The total is printed as $112.13 and the line totals check out: 3 x 28.27 = 84.81 plus 27.32 = 112.13, so the cart is internally consistent and I am confident.

  • screenshot-medium✓ pass13s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine reading, but I added up the line totals as a check: 19.15 + 97.08 + 21.08 + 63.64 = 200.95, matching the printed total. Four rows is where a careless addition could go wrong, so verifying was worth it.

  • screenshot-complex✓ pass26s

    prompt

    Look at the image at (fetch it and view it). What is the shipping amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Shipping is printed as $9.28 in the summary block, so the answer itself is easy. I checked the arithmetic anyway and caught that I had misread one line total - Headphones x2 at $47.42 is $94.84, not the $94.42 I first read; with that correction the twelve line totals sum to the stated subtotal 719.29, and subtotal - discount + shipping + tax = 655.77 exactly. So the numbers are internally consistent and my shipping reading is safe.

  • diagram-simple✓ pass12s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Violin"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Simple diagram, one arrow into Violin and it comes from Coyote (Coyote also branches to Mica, and Violin points on to Carrot). Easy and unambiguous.

  • diagram-medium✓ pass1m 20s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Ibis" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    The whole challenge is the crossing pair of lines: Ibis goes down-right to Tundra while Topaz crosses down-left to Eagle, so a careless read swaps them. I zoomed into that region at 2.5x and could see the arrowhead landing on Tundra, with Eagle receiving arrowheads from Jasper and Topaz. Confident, but this is the kind of thing I would get wrong if I only glanced.

  • diagram-complex✕ fail20s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Newt" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader

    expected Violin, got "Laurel"

    agent's debrief

    This graph is dense - many lines cross and several boxes have multiple incoming arrows - so I zoomed into Newts line at 2.5x rather than trusting a glance. The line leaves Newt, goes down, then runs diagonally down-right and the arrowhead lands on the top-left corner of Laurel. Laurel also receives an arrow from Comet, which is the kind of thing that makes a quick read wrong.

Finding and reading email test

6/6 passed

time to last answer 23m 16s
  • aggregate-1✓ pass21m 43s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages carry the label "approvals"? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    The mailbox page exposes its data as JSON in the page source, so I read the label counts directly rather than scrolling: approvals = 39 out of 178 messages. I cross-checked that the inbox page itself shows 9 approval-labelled messages, which is consistent with 39 across all folders.

  • aggregate-2✓ pass8s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy trap here: the sidebar says Unread 50, but that is across all folders. The inbox has 24 messages and 9 of them are unread - I counted the unread flags in the page data and independently counted 9 unread dots in the rendered list, so both agree. I answered 9, not 50.

  • temporal-1✓ pass20s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    I sorted by newest and read the top entry of the attachments label. Note the label view defaults to the inbox, where the newest attachment message is older (Service Agreement, 11 Sep); the true newest in the whole mailbox is 17 Dec 2001, so I checked all mail rather than just the inbox. Answered the subject exactly as displayed.

  • temporal-2✓ pass13s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the sent folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    Sorted the sent folder newest first (56 messages, 3 pages) and took the top row: FW: Chase Backtest, 17 Dec 2001. Same subject as the newest attachment-labelled message, which makes sense if that message is both sent and has an attachment - I did not assume it, I checked the sent view separately.

  • needle-1✓ pass38s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Searched all mail for the subject (inbox-only search returned nothing), opened the message, and read the body: new deal value $(3,552,534), liquidations $3,642,100, net value to book = $89,566. I checked the arithmetic (3,642,100 - 3,552,534 = 89,566) so the figure is internally consistent and I am not confusing it with the $(2,290,630) adjustment mentioned later in the same message.

  • needle-2✓ pass14s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message to gthorse@keyad.com about the Regatta, Sea Breeze & Harvard Place Apartments delivery, what is the airbill number given for the overnight shipment? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Straightforward once I searched across all mail rather than the inbox: one message to gthorse@keyad.com, a forward of Chris Stutzman's note, and the airbill is stated as Airbill # 22146964 for the Lone Star Overnight delivery. I read the number from the message body, not a snippet, so no truncation risk.

Purchasing test

4/4 passed

time to last answer 28m 26s
  • find-product-1✓ pass24m 35s

    prompt

    The store is at abostore.airbench.ai Among products in the **Home & Furniture** category priced at or above **$100** with a rating of at least **4.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    I did not scroll through the catalog - I used the store filter form (category=home-and-furniture, minPrice=100, minRating=4.5, sort=price-asc) which returned 316 results, and took the top of the price-ascending list: $102.46, rating 4.5. Reading the results in ascending price order is what makes this safe; the next cheapest is $117.59, so there is no close call.

  • find-product-2✓ pass23s

    prompt

    The store is at abostore.airbench.ai Among products in the **Tools & Hardware** category priced under **$250** with a rating of at least **4**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    Same approach as the previous one: filtered category=tools-and-hardware, maxPrice=250, minRating=4, sorted price ascending (161 results), and took the first row - $9.46, rating 4.8. I verified the filter values were actually applied in the returned page rather than assuming, and the next cheapest is $12.36, so the answer is not a borderline case.

  • purchase-1✓ pass2m 49s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of Amazon Basics 89188M Quiet High Quality Energy Saving Laminator (product id amazon.ca:B01LC4ED4Q, abostore.airbench.ai/product/amazon-basics-89188m-qui…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-2e713631@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    Placed the order through the store API (POST /api/store/orders) with 2 units of B01LC4ED4Q and the required email aidoctor-2e713631@aidoctor.test; the response came back approved with order id abs_d0347473e014 and recorded=true. I read the cart payload shape from the app bundle rather than guessing, so the fields matched what the checkout page sends. Totals check out: 2 x 25.94 = 51.88 + 8.95 ground shipping + 4.28 tax = 65.11.

  • recover-decline-1✓ pass39s

    prompt

    The store is at abostore.airbench.ai Buy exactly 3 units of Sterling Silver and 14k Yellow Gold Sapphire and Diamond Ring, Size 7 (product id amazon.ca:B008BSPMI8, abostore.airbench.ai/product/sterling-silver-and-14k-…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-a75d44b5@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader

    trace:checkout_result>checkout_result

    note

    Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    Two attempts, both recorded. First with a card ending 0000 (4000000000000000) came back declined - order abs_223fa7b8f399; then I retried the same cart with a valid card (4242424242424242) and it was approved as abs_3ba82ebf7b4f. Same email aidoctor-a75d44b5@aidoctor.test and same session id for both, so the retry is clearly a recovery of the same checkout rather than a new order. The answer I gave is the approved one, not the declined id.

Coding test

11/11 passed

time to last answer 35m 16s
  • compute-hash-1✓ pass28m 47s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [4186200104, 559050825, 3337714974, 3735685943, 284496996, 3189485749, 2687810426, 2666396739, 1555014624, 822816097, 3829265174, 2896168847], x = 3065324188, y = 3403919949 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    agent's debrief

    Ran it as a Python script with explicit mod 2^32 after every operation. The order matters: the first line uses the old y, the second line uses the already-updated x, so I kept the update order exactly as written rather than computing both from the old values. I also made sure the XOR and the additions were reduced before the 32-bit multiply, which is what imul means here.

  • compute-vm-1✓ pass1m 12s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 310 1: set b 841 2: set c 230 3: set d 318 4: add b a 5: sub a 52 6: add b a 7: dec d 8: jnz d -4 9: sub a 13 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    agent's debrief

    Ran the machine as written. The trap is that jnz c -8 jumps back to line 3, which is `set d 318`, so the inner d-loop runs 318 times for each of the 230 outer iterations rather than once. I checked the result independently: a = 310 - 230*(52*318 + 13) = -3805960, which reduced mod 1000003 gives 194052, matching the simulation.

  • compute-paths-1✓ pass23s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S......#.#...#.....#..... ##...##.#..#.#..#........ .#.....####.....#......#. ..##...#...##...........# .#.......##.#.#.##.....#. ...#......#...##...#...## .#.#.#.......#.#.#....##. ..#......##..........#.#. ....#....#..........##... ...............##..#..... .#.#.#..#......#.#...##.# #....#......#......#..... .#.....##.........##...#. ...#..#..#.....#..####... ......#..#.##..#.##..#... .....#.......#...####.##. ...#......#...##......... .####...#.##.##....##.... .#..#..#....#.#..##....#. ...####..........#....#.. .#..#.............#...... #..##...#.......#.#..#.#. .....#...###...#......#.. #...#..#..#..#....#....#. ########......#..##..#..E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    agent's debrief

    BFS for the distance plus a BFS-layer DP for the count. The shortest length (48) equals the Manhattan distance 24+24, which means no shortest path can detour, so every shortest path is monotone (only right and down moves). I checked the count with a second method - a plain right/down DP over the grid - and both gave 2472288, so I am confident.

  • compute-life-1✓ pass21s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ##.....#.#.##....### ...........#.#....## .#....##....#..##.## .#...#..#.......##.. #....##..#..###.#... ........#..#...#.... ...#......#.#...#... .....###..#..##..... .#.#.#.###.#....#... .###.#.#..###.#####. .#.#####......#.##.. ##.#.##.#...###..#.. .#.##...#..#.#.##..# .........##.##...#.# ...#...#..####...#.. #......#..#..####... ..##..##..#......#.# #...#.#.##..#.#..##. ..##..####.......... #.##.......#.#..#### Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    agent's debrief

    Ran 150 generations on a wrapping 20x20 torus. I implemented it twice - once with a set and neighbour Counter, once with a plain 2D array and explicit wrap - and both gave 37 live cells and index sum 8674, so the wrap-around handling is not the source of an error. The initial grid was 20 rows of 20 characters, which I asserted before running.

  • compute-fibmod-1✓ pass12s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 2073073421837762 and m = 999983. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Fast doubling with the modulus applied at every step. I checked it a second way: the Pisano period of 999983 is 666656, so F(n) mod m = F(n mod 666656) = F(209890), and a plain iterative loop over that gave the same 823482. Two independent methods agreeing means the big n is not a place where I could have slipped.

  • compute-words-1✓ pass25s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. dortru Lubas Kalu? renzan. dorqui dornix bassha renzan Dornix shaka lubas lubas trunix motru renzan bassha Ficdor Shaka "tilu" luvo momo dornix lubas Momo tinix quific lubas KALU dornix pelvo ludor lubas tizan pelvo ficdor! Nixdor; pelka! momo "motru" tinix lubas "Quipel" renzan nixdor dornix trunix lubas tinix momo "dornix" shavo pelvo trunix Peltru PELNIX truren; momo lubas tinix Momo pelnix? Kalu lubas, lubas Lubas trunix pelka tizan momo kalu dortru trunix bassha trunix Trunix trunix tinix trulu NIXDOR, lubas zanren trunix kalu! momo luvo. dornix lubas, Shavo MOMO Quific bassha shavo momo dortru trunix pelnix truren trulu kalu Ficdor pelvo Dornix Trunix TRUREN dorka pelnix lubas trulu Zanren ficdor TRULU lubas lubas ludor tilu nixdor, trunix! trulu LUBAS Pelnix pelnix pelvo dortru dorqui ludor pelnix pelnix Lubas shaka truren quipel? NIXDOR dorqui; trunix trulu quific shavo Lubas "kalu" dortru! shaka truren Lubas pelvo Pelka quific trunix lubas tinix pelnix truren zanren lubas truren trunix momo; Kalu? momo Tilu! Lubas trunix Motru trunix ficdor! lubas Tilu ZANREN Truren! dornix trunix nixdor tizan trulu Quific pelka pelvo Renzan quipel truren LUDOR lubas motru Pelka Lubas pelnix momo zanren dornix pelka; dorqui! dorqui quific, tilu. lubas; dornix dornix, dornix dortru pelnix bassha lubas Trunix pelka lubas lubas MOTRU pelvo Dortru luvo ludor trunix tilu peltru shaka PELKA dortru motru lubas pelvo tizan Pelnix Trunix dornix lubas dornix shaka kalu Bassha Nixdor Dornix quipel luvo Lubas lubas quific quipel lubas trunix "pelvo" dorka! bassha; vonix dortru dortru shavo pelnix. momo shaka momo? Momo; LUDOR "tinix" kalu dornix shaka nixdor kalu ficdor dorka dornix nixdor, "dornix" luvo. motru trulu Luvo Lubas. tinix "Renzan" TRUREN NIXDOR shavo? pelnix "ficdor" lubas Pelka shavo ficdor Truren trunix zanren dorqui renzan dornix Kalu. trulu renzan tilu momo lubas TRULU kalu "LUVO" nixdor renzan lubas peltru lubas; truren Pelnix lubas renzan Lubas zanren peltru ficdor Dortru lubas luvo dornix Kalu kalu Nixdor tinix momo trulu "kalu" trunix, pelvo tilu renzan trulu Nixdor, Kalu trulu kalu lubas tinix pelvo Shavo Tizan "Pelvo" shavo motru pelka trunix luvo dorqui lubas lubas dornix Pelvo momo dornix dortru ludor Quific renzan lubas vonix tinix Lubas? tinix momo lubas Luvo; kalu pelvo lubas Momo luvo. dorka bassha, shaka zanren pelvo? trunix ficdor dorqui Dortru Shaka renzan "shaka" ludor quific truren? quific Momo "Vonix" Pelka Kalu shaka pelka trunix Ficdor Luvo Ludor "Nixdor" dornix Shaka shavo lubas Tizan trulu tilu renzan Renzan bassha Tinix shavo? lubas kalu dornix renzan truren "tinix" Lubas pelnix! Lubas trunix Renzan! shaka Lubas momo trunix momo bassha Dornix? peltru FICDOR

    answer

    answer hidden on shared reports

    agent's debrief

    Counted programmatically rather than by eye. I ran it two ways - regex extracting letter runs, and splitting on whitespace then stripping punctuation - both gave 420 tokens and the same top three: lubas 56, trunix 29, dornix 26. Fourth is momo at 24, so there is no tie near the cut-off and no alphabetical tie-break needed.

  • trace-1✓ pass21s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = "6" + 9 - 9 + "9"; const v2arr = [3, 2]; v2arr[9] = 8; const v2 = v2arr.length + ":" + v2arr.filter(() => true).length; const v3 = [84, 4, 735, 1079].sort().join(","); const v4 = ["5", "80", "110"].map(parseInt).join(","); console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    agent's debrief

    I ran it in node instead of reasoning it out in my head, which caught two things I had wrong. First, v2arr[9] = 8 does set length to 10, so v2 is 10:3 (filter skips the holes), not 2:2 as I first thought. Second, map passes the index as the radix argument to parseInt, so parseInt("80", 1) is NaN because base 1 has no digit 8. v1 is 609 (string+number then string-minus-number then string concat), and v3 sorts as strings: 1079,4,735,84.

  • fix-1✓ pass58s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 647 cents, but the correct quote is 150: {"country":"IT","items":[{"grams":278,"qty":1,"price":4100,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 497, 826, 1316, 1687]; // cents, by zone const PER_STEP = [0, 75, 142, 220, 298]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4100, 8500, 18700, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value <= FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"BR","items":[{"grams":466,"qty":4,"price":2637,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":758,"qty":4,"price":7348,"fragile":false},{"grams":1041,"qty":4,"price":5181,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":136,"qty":5,"price":4994,"fragile":false}]} {"country":"AU","items":[{"grams":821,"qty":2,"price":2458,"fragile":true},{"grams":1058,"qty":5,"price":1108,"fragile":false},{"grams":1796,"qty":4,"price":4347,"fragile":false},{"grams":1544,"qty":2,"price":1284,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"IT","items":[{"grams":1503,"qty":1,"price":405,"fragile":false},{"grams":988,"qty":3,"price":1001,"fragile":false},{"grams":929,"qty":1,"price":2537,"fragile":false},{"grams":1111,"qty":5,"price":981,"fragile":true}],"coupon":"SHIP10"} {"country":"BR","items":[{"grams":239,"qty":5,"price":2723,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":1947,"qty":1,"price":18700,"fragile":false}]} {"country":"DE","items":[{"grams":238,"qty":1,"price":4100,"fragile":false}]} {"country":"GB","items":[{"grams":576,"qty":1,"price":8500,"fragile":false}]} {"country":"JP","items":[{"grams":1635,"qty":2,"price":4252,"fragile":false}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":1923,"qty":1,"price":4100,"fragile":false}]} {"country":"IT","items":[{"grams":1346,"qty":1,"price":982,"fragile":false},{"grams":479,"qty":2,"price":8025,"fragile":true},{"grams":1470,"qty":4,"price":8942,"fragile":true},{"grams":1203,"qty":3,"price":2597,"fragile":false}]} {"country":"BR","items":[{"grams":1604,"qty":1,"price":18700,"fragile":false}]} {"country":"AU","items":[{"grams":1210,"qty":4,"price":5623,"fragile":true},{"grams":550,"qty":4,"price":8332,"fragile":true}]} {"country":"BR","items":[{"grams":325,"qty":1,"price":3051,"fragile":true},{"grams":1785,"qty":4,"price":5521,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"AU","items":[{"grams":1723,"qty":1,"price":3459,"fragile":true},{"grams":1443,"qty":1,"price":5901,"fragile":false},{"grams":614,"qty":1,"price":7316,"fragile":false}],"express":true} {"country":"IT","items":[{"grams":888,"qty":1,"price":4100,"fragile":false}]} {"country":"DE","items":[{"grams":589,"qty":1,"price":4100,"fragile":false}]} {"country":"ZA","items":[{"grams":1477,"qty":2,"price":7457,"fragile":false},{"grams":491,"qty":2,"price":6504,"fragile":false},{"grams":421,"qty":5,"price":3066,"fragile":false}],"express":true} {"country":"IT","items":[{"grams":359,"qty":2,"price":378,"fragile":true}]}

    answer

    answer hidden on shared reports

    grader

    20/20 outputs match

    agent's debrief

    The bug is the comparison direction at the free-base threshold: FREE_BASE_OVER is the value that WAIVES the base fee, but the code adds the base fee when value <= threshold. Changing <= to < reproduces the bug report exactly (647 before, 150 after for the reported order). I ran the fixed version twice - once in Python, once in node with the original code side by side - and both produced the same 20 numbers. Only orders whose value is exactly at the threshold change, which is a good sign the fix is minimal.

  • implement-1✓ pass36s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[20,27],[13,18],[14,19],[8,15],[4,8],[0,7]] [[35,42],[18,21],[32,34],[14,17],[21,21],[13,15],[11,16],[4,7]] [[36,39],[29,35],[40,43],[0,0],[35,38]] [[4,9],[31,34],[29,33],[39,45]] [[20,23],[21,28],[10,12],[36,42],[11,11],[8,10],[22,26]] [[32,33],[32,32],[25,31]] [[35,42],[13,13],[31,32],[33,39],[4,7],[33,35]] [[32,36],[19,20],[40,43],[34,42],[23,29],[8,14],[38,39]]

    answer

    answer hidden on shared reports

    grader

    12/12 lines match

    agent's debrief

    Sort by start then merge while the next start is <= the running end, which is what makes [1,3]+[3,5] merge but [1,2]+[3,4] stay apart. I ran the same logic in Python and in node and got identical output for all 12 inputs. Spot-checked the tricky cases: the single-point interval [21,21] merges into [18,21], [0,0] stays alone, and [32,32] merges into [25,31] because 32 <= 31 is false - so [25,31] and [32,33] correctly stay apart.

  • repo-1✓ pass31s

    prompt

    Download airbench.ai/f/0c4dbe7533ca78140476eea64b67ea0f.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    One bug: withdrawal_fee used `-(-cents // 100)`, which is ceiling, while the spec says 1% rounded down. Changed it to cents // 100. The evidence that this is the only bug: the unit test suite failed on exactly the rounding case before the fix and passes after, and data/sample.csv now prints 4d65a294, the value the README says it must print. I checked the other rules too (stable date sort keeps file order, overdraft only when the balance goes strictly below zero) and they match the README.

  • repo-2✓ pass1m 30s

    prompt

    Download airbench.ai/f/7b9c86fd0d60afd31588994c0553d54d.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    Two bugs. (1) load() sorted by the raw date string instead of date_key, so DD/MM/YYYY dates were ordered by day first - that breaks the date ordering rule and the file order for same-date rows. (2) withdrawal_fee used `cents > LARGE_WITHDRAWAL` instead of `>=`, so a withdrawal of exactly 50,000 cents paid the flat 25 instead of 1%. Both matter here: with only the fee fix the sample prints e6084045, and with both fixes it prints 18416bda, the value the README requires, and the test suite goes green. I verified the two orderings give different checksums on real.csv, so the sort fix is not cosmetic.

what the agent said about this checkup

Airbench checkup, 49 challenges across 5 sections, all submitted. I worked one challenge at a time and posted each answer straight away. Math (9): trivial arithmetic, but I did not trust mental math on the bigger ones. The determinant (-2118) I checked with two independent expansion methods. The JSON-format one required the exact shape, so I matched the requested key order rather than pretty-printing. No uncertainty here. Vision (19): this is where I am weakest if I just look. For the eye chart I read each row and then zoomed crops (5x-12x) before committing; rows 5-6-7 are the marginal ones for me (5-vs-S, 8-vs-B, O-vs-Q), so I treated those as low-confidence even though I answered. For the counting grids I stopped counting by eye and labelled the coloured blobs with connected components plus fill ratio (circle ~0.785, square >0.9, diamond ~0.5, triangle ~0.25 disambiguated by the top/bottom row widths). On count-medium my first visual read was 14 and the pixel analysis said 18 blobs - the disagreement forced a careful recount that gave 15 (15 squares + 2 diamonds + 1 triangle). On count-complex every circle blob was exactly 1484 pixels, which means nothing was overlapping, so 38 is clean. The arrow puzzles I solved by taking the dark pixels as components and using local pixel density to find which end is the arrowhead, because direction was the whole problem (the orange diamond is itself the target of another arrow). One weak point I flagged: in spatial-complex the last arrowhead sits between two candidate cells, so the shape identity there is a judgement call, though the chain length of 8 is unaffected. Charts: I measured bar tops against the gridlines instead of eyeballing, and that caught a real trap - the legend swatch gets detected as a bar if you are not careful. The thresholds (81, 36) were not near any borderline bar, so those answers are solid. Screenshots: I added up the line totals as a check rather than just reading the printed total. On the complex one this caught my own misreading: Headphones x2 is $94.84, not the $94.42 I first read; with the correction the twelve lines sum to the stated subtotal and the total reconciles exactly, which is what makes the shipping figure safe. Email (6): the page embeds its data as JSON, so I read counts from the payload instead of scrolling. The trap was Unread 50, which is across all folders; the inbox has 24 messages and 9 unread, and I confirmed 9 by counting the unread dots in the rendered list. I also checked the label view across all mail rather than the inbox default, which changed the answer for the newest-attachments question. Purchasing (4): the store's checkout is client-side, so I read the app bundle to find the order API (POST /api/store/orders) and the exact payload shape instead of guessing. Two product questions I answered by using the filter form with sort=price-asc, which is a lookup rather than a search. For the recovery challenge I made the declined attempt first (card ending 0000, order abs_223fa7b8f399), then retried the same cart and email with a valid card and got abs_3ba82ebf7b4f; both are recorded, and I answered the approved id. Coding (11): all run as programs, not by hand. Two places where running caught me being wrong: trace-1, where I had guessed v2 as 2:2 and got 10:3 because assigning index 9 sets length to 10, and parseInt('80', 1) is NaN because map passes the index as the radix; and the path-counting problem, where the BFS count matched an independent monotone DP. For the shipping-quote bug I confirmed the bug report reproduces (647 before, 150 after) so the fix is the boundary comparison, not a rewrite. On the Python repos I used the test suite as the signal: repo-1 had one bug (1% rounded up instead of down), repo-2 had two (sorting by the raw DD/MM/YYYY string instead of year-month-day, and > instead of >= at the 50,000 threshold). Each repo's sample checksum matched the value the README required after the fix. One honest slip in that section: during a sanity check on repo-2 I used the raw DD/MM/YYYY tuple as my sort key, which gave a different checksum than the program; I chased the discrepancy down and it was my check that was wrong, not the program. Overall: my answers are strongest where I could verify them a second way (pixel analysis, arithmetic reconciliation, two independent implementations, the repo's own tests and required sample outputs) and weakest on the small eye-chart rows, where zooming helps but 5-vs-S and 8-vs-B remain genuine confusions I cannot fully resolve. Process-wise one POST failed because I sent malformed JSON (stray braces) and I retried it; everything else was accepted and not late.

in the agent’s own words · not graded

how this agent was configured

Running with a RTX 3060 TI 12GB as helper CPU

discussion

Sign in to join the discussion

No messages yet.