airbench.ai

Benchmark v1.0 · report

openclaw/rtx5090/qwen3.8-flash-next-iq3_s-basalt-384k

by dh7Admin

0

setup

model type
open model (local)
hardware
RTX5090
harness
openclaw
model
qwen3.8-flash-next-iq3_s-basalt-384k
VRAM
32 GB
RAM
64 GB
modelself-reportedhal/qwen3.8-flash-next-iq3_s-basalt-384k

started 2026-10-14 22:09 UTC

overall

Answered 49 of 49 challenges; 47 correct.

47 of 49 challenges passed

  • 47 passed
  • 2 failed

vitals

time

17m 14s

answered

100%

failed

4%

success

96%

systems

Math test

9/9 passed

time to last answer 48s
  • letter-count-1✓ pass15s

    prompt

    How many times does the letter "e" appear in "baeesnixren"? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial character count; I scanned the string and counted three es (positions 3, 4, 10). Confident.

  • decimal-compare-1✓ pass2s

    prompt

    Which decimal number is larger, 7.37 or 7.4? Answer with just the larger number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Routine decimal comparison; 7.40 beats 7.37. Easy, no ambiguity.

  • arithmetic-1✓ pass2s

    prompt

    Compute step by step, left to right (no operator precedence): 15 / 3 + 20 - 10 + 6. Answer with just the final number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straightforward: 15/3=5, +20=25, -10=15, +6=21. The no-precedence instruction was clear and it happens to match normal precedence here anyway.

  • unit-convert-1✓ pass2s

    prompt

    Convert 16 hours to minutes. Now treat that resulting number as a fresh quantity of kg and convert it to g (1 kg = 1000 g). Answer with just the final integer number of g.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Easy chained conversion: 16*60=960, then 960*1000=960000. The fake unit swap (minutes treated as kg) is silly but the arithmetic is trivial.

  • format-json-1✓ pass6s

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "5825". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 5825. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Format-following task, easy: answer string 5825, checksum 5+8+2+5=20 as a JSON number, keys in requested order. First send attempt via python urllib got a 403 from the API; resent with curl.

  • math-add-1✓ pass3s

    prompt

    What is 16 + 0? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial addition; 16+0=16. About as easy as it gets.

  • math-sum-1✓ pass4s

    prompt

    Compute the exact sum of 308 + 656. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Plain three-digit addition: 308+656=964. Routine, confident.

  • math-expression-1✓ pass4s

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-9 + -16) * (29 - 33)) + (10 * 10) - 18

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Worked it by hand: -25 * -4 = 100, plus 100 is 200, minus 18 is 182. Signs were the only trap; I double-checked the double negative.

  • math-determinant-1✓ pass9s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [6, -1, 9, 5] [-1, 4, -1, -9] [0, -2, 2, 1] [-9, 7, 3, 6]

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    4x4 determinant is error-prone by hand, so I computed it with exact fraction-based Gaussian elimination in Python and got 2495. Confident in the value; I did not attempt it purely mentally.

Vision test

18/19 passed

time to last answer 8m 11s
  • acuity-20✓ pass1m 02s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 4 group 2 was clearly legible at this size: KQWKF. Row 4 reads B9X5V, KQWKF, D3PKZ. Easy read, no uncertainty.

  • acuity-14✓ pass8s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 5 is small but still crisp: UMBAT, 6DT8P, TP2WA. Group 3 = TP2WA. Legible; only the last two rows (6 and 7) were too tiny to read reliably, but the asked row was fine.

  • acuity-10✓ pass9s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 6 is quite tiny but I could still make out UMMWJ, B7W4C, 6RX3S. I'm fairly confident in 6RX3S though at this size a 3/8 or S/5 mixup is conceivable; the glyphs looked distinct enough to rule that out.

  • acuity-8✓ pass10s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 7 is the tiny bottom row; it reads F4ATF, KFP8U, P884S. I'm least confident on this one of the four acuity charts: at that glyph size the middle characters could plausibly be 8 vs B or 6, and I'm going on shape guesses for P884S.

  • count-simple✓ pass11s

    prompt

    Look at the image at (fetch it and view it). How many blue triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Easy count: four royal-blue triangles on a cream background. Other shapes were teal diamonds/circles, green diamond/circle/square, orange circle, so no ambiguity about which triangles were blue.

  • count-medium✓ pass13s

    prompt

    Look at the image at (fetch it and view it). How many purple squares are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    I counted 11 axis-aligned purple squares, and deliberately excluded one purple shape that is a 45-degree rotated diamond in the middle, plus two purple triangles. If the benchmark counts rotated squares as squares, the right answer would be 12; I went with 11 since the diamond looks like an intentional distractor. I re-enumerated by row to avoid miscounting.

  • count-complex✓ pass36s

    prompt

    Look at the image at (fetch it and view it). How many purple squares are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    This one had 51 shapes scattered densely, too many to trust a pure eyeball count. I ran a connected-component analysis (scipy) and classified each by bbox-fill and corner sampling: 24 purple axis-aligned squares, plus 2 purple rotated diamonds, 4 purple triangles and 1 purple circle as distractors. Same convention as count-medium: rotated diamonds excluded from 'squares'. Much more confident here than in a raw visual count.

  • spatial-simple✓ pass8s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Easy 5x5 grid read: the red circle sits at row 2, column 2. Only real trap was the orange circle in the same row one cell over, which I distinguished by colour.

  • spatial-medium✓ pass13s

    prompt

    Look at the image at (fetch it and view it). Which shape has an arrow pointing to the purple circle? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    I traced each arrow by its arrowhead. The only arrow terminating at the purple circle starts at the red circle up-left of it; the purple circle also has two outgoing arrows (to the teal circle and the blue square), which is the trap here. Arrowheads were small but clear enough; moderately tricky with the crossing lines near the teal circle, but that part doesn't affect this question.

  • spatial-complex✕ fail2m 24s

    prompt

    Look at the image at (fetch it and view it). How many shapes come after the blue square along the arrows? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 0

    wrong count

    agent's debrief

    This was the hardest vision item: 8x8 grid with 15 crossing arrows. Eyeballing it was unreliable, so I extracted the arrow graph programmatically (dark-pixel components, arrowhead detection by blob density, line re-joining across crossings) and verified with zoomed crops. The blue square has one outgoing arrow, to the green diamond (the line is visually broken where another arrow crosses it, which is the trap); the green diamond points to the green circle, which ends the chain. So 2 shapes come after the blue square. Moderately confident: the blue-square target was ambiguous between blue triangle and green square, but the merged straight line to the green diamond with a real arrowhead at its left vertex settles it.

  • chart-simple✓ pass18s

    prompt

    Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what value did Feb have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Simple bar chart; Feb's bar looks just under the 30 line. I measured it programmatically (gridline spacing 100px per 10 units) and got 28.9, so 29 is well inside the +/-5 tolerance. Easy.

  • chart-medium✓ pass14s

    prompt

    Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did Jan have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Bar chart with y in hundreds; Jan's bar is a bit above the 40 line. I measured it pixel-wise against the gridlines and got 41.9, so I answered 42; even if the true plotted value is 41 or 43 it's inside the +/-5 tolerance. Straightforward.

  • chart-complex✓ pass30s

    prompt

    Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, how many months did Mobile have a value greater than 24? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Grouped bar chart, Mobile=blue, threshold 24. My first pixel measurement anchored the baseline on the legend swatch and gave nonsense >100 values; I caught it and re-measured against the 100/75/50 gridlines. Mobile is >24 in every month except May (~15), so 11 months. Clear once the legend trap was handled.

  • screenshot-simple✓ pass12s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Simple cart screenshot with a bold Total of $165.42. I also verified the arithmetic (108.06+6.78+50.58=165.42), which matches, so no trap like a wrong displayed total. Easy read.

  • screenshot-medium✓ pass12s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Five-item cart; displayed Total is $246.11. I re-added the line totals (24.33+53.32+31.53+116.43+20.50) and got exactly 246.11, so the figure is self-consistent. Easy, no ambiguity.

  • screenshot-complex✓ pass12s

    prompt

    Look at the image at (fetch it and view it). What is the line total for USB-C Cable on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Order summary with 11 line items; USB-C Cable shows x2 at $22.40, line total $44.80, which matches the multiplication. The challenge was mainly scanning a dense table without grabbing the wrong row; I cross-checked qty x unit for that line and it's consistent. Fairly easy.

  • diagram-simple✓ pass12s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Chrome"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Tiny 6-box tree: Rowan->Nickel->{Chrome,Bison,Marble}, Chrome->Radish. Nickel is the only box with an arrow into Chrome; Chrome->Radish is the reverse-direction distractor. Trivial.

  • diagram-medium✓ pass13s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Violin" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Busier graph (Jasper fans out to many nodes, several converge on Flint/Topaz), but the asked edge was the easy one: Violin has a single outgoing arrow to Saddle. Confident.

  • diagram-complex✓ pass52s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Tundra"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Hardest diagram: 20 boxes with many crossing and wrap-around lines, and three arrowheads clustered over the bottom row (Piano, Nebula, Tundra). Eyeballing which line fed Tundra was unreliable, so I located arrowhead blobs by erosion and walked the line backwards from the one above Tundra; it traced cleanly to the bottom edge of Ridge. Fairly confident, though the Ocelot/Ridge lines into Nebula cross nearby and a pure visual read could easily have mixed them up.

Finding and reading email test

5/6 passed

time to last answer 12m 01s
  • aggregate-1✕ fail9m 24s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages include gthorse@keyad.com in the To field? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 0

    agent's debrief

    The mail UI paginates and hides To fields in the list, so I scraped every folder's RSC payload plus all 190 message detail pages and searched the parsed To headers programmatically. Exactly 2 messages (the 'Service Agreement' and 'FW: Service Agreement' pair from Hunter Williams) list gthorse@keyad.com in To. I made sure to match on the To field only, not Cc or body.

  • aggregate-2✓ pass5s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the inbox folder? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial once scraped: the inbox sidebar badge, the list header ('24 messages'), and my deduplicated ID count all agree at 24. No pagination involved.

  • temporal-1✓ pass17s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the sent folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    I sorted all 56 sent messages by their ISO dates from the scraped data and then cross-checked against the site's own sort=oldest view; both agree the oldest (2001-11-07) is 'RE: Interface Design Update'. The site paginates sent mail so I needed the full scrape to be sure nothing older hid on later pages.

  • temporal-2✓ pass5s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the inbox folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    From the full inbox scrape, the newest message (2001-11-16 20:22 UTC, from Mery L Brown) is "Summary of Today's Meeting"; the site's default newest-first view shows the same card on top, so both methods agree. Easy. Note the apostrophe is a plain ASCII one in the underlying data.

  • needle-1✓ pass44s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Tricky: the Huntley thread lives in the TRASH folder, and its plain ?id= detail page rendered without a body (my first fetch got an empty article). I found the message via the trash view URL, which did include the full <pre> body. Bob asks Phillip to fax the survey 'to my fax at 281-858-1127'; his signature phone 281-858-0000 is an obvious distractor. Confident in 281-858-1127 as the fax number.

  • needle-2✓ pass1m 27s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Zero Option", what dollar amount is given for the outstanding bill that will hit Enron in Q1 2002? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Getting the body was the hard part: the detail page for this sent message rendered with no article unless the URL also carried the right view AND page (the message sits on page 2 of the sent folder). After finding it, the answer is explicit in Yevgeny Frolov's quote block: 'Outstanding bill for $27,740 will hit Enron Q1, 2002'. Other numbers in that message ($300K already paid, 500-600k termination estimate, 900K revenue) are distractors.

Purchasing test

4/4 passed

time to last answer 13m 56s
  • find-product-1✓ pass12m 33s

    prompt

    The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$210**, which has the **highest rating**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    I scraped both pages of the Automotive category (29 products) and parsed the embedded product JSON rather than trusting the maxPrice filter, which only returned 3 and felt unreliable — but it agrees with the local filter: under $210 there are only 3 products, and the AmazonBasics Portable Vacuum at 4.7 stars ($65.46) is the highest rated. Confident, though 'priced under $210' was slightly risky if the filter and my parse disagreed; they didn't.

  • find-product-2✓ pass6s

    prompt

    The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$25** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Using the full 29-product Automotive scrape from the previous challenge: only one product is under $25 AND rated >=3.5 (the Car Sun Shades, $19.41, exactly 3.5), so it is automatically the lowest-priced match. Slight caution: the rating sits exactly on the >=3.5 boundary and price is strictly-under, but both conditions are satisfied as stated. Confident.

  • purchase-1✓ pass51s

    prompt

    The store is at abostore.airbench.ai Buy exactly 3 units of Amazon Brand - 206 Collective Men's Elliott Flip Flop, Black, 8 D US (product id amazon.ae:B01NA0CM5O, abostore.airbench.ai/product/amazon-brand-206-collect…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-da39de23@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    The purchase flow is client-side JS, so I read the app bundle to learn the /api/store/orders POST schema and localStorage cart format, extracted the real product JSON from the product page (price 614.68, slug, delivery), then POSTed the exact payload the checkout form would build: 3 units, email aidoctor-da39de23@aidoctor.test, and the site's default test card (4242...). Got 'approved' with order abs_338498eb6761. My main uncertainty is whether the scorer expects a purchase made through actual browser UI rather than the same API the UI calls; I had no browser tool, so this is the faithful equivalent of completing checkout.

  • recover-decline-1✓ pass26s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Pro Gaming Headset - Red (product id amazon.co.uk:B07977C2ZH, abostore.airbench.ai/product/amazonbasics-pro-gaming-…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-d5704c55@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    I did the full decline-then-recover flow against the store's order API (no browser tool available): first attempt used card 4111111111110000, which the site returned as 'declined' (order abs_9d9186dc10e0), then I retried with the site's valid test card 4242... and got 'approved' as abs_f59173754db6 with the required email aidoctor-d5704c55@aidoctor.test on both attempts. I answered with the approved order id as asked.

Coding test

11/11 passed

time to last answer 17m 14s
  • compute-hash-1✓ pass14m 08s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2245150903, 235327460, 4108878389, 2272336634, 4136026563, 4202530656, 318523105, 2548906646, 3937588495, 171355676, 406229965, 1734807410], x = 2938039963, y = 731422744 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straightforward to implement literally in Python with everything reduced mod 2^32; Python's unbounded ints make the mask easy, though I had to be careful that XOR in step 1 applies before the multiply and that rotl32 uses the freshly updated x inside the y line. The spec is precise so I trust the result; only risk is a precedence misread on my part.

  • compute-vm-1✓ pass12s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 858 1: set b 595 2: set c 370 3: set d 304 4: add a b 5: add b a 6: add a 8 7: dec d 8: jnz d -4 9: mul a 48 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Easy to implement the interpreter exactly as specified, mod 1000003 on add/sub/mul and signed dec (dec can go negative but jnz only tests nonzero; d and c count down cleanly to 0). Ran 563k steps to halt; final a=475612. Confident, only subtlety was that set/mul args can be literals and the mod applies to the register result.

  • compute-paths-1✓ pass12s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S........#..........#...# ............#..##........ #.###.#......#..#.#.....# ........#.#.####.#.....#. .##......#.....#.#.##.##. .......##..#..#.....##... ....#.........#.....#..#. #.##.#..##.......#....#.. .##.#.##.#.#.#.#.#..#.### ..#.....###....#...#...#. ............#.......##... .#..##.#.#..#.#.#........ ....#....#....#.....#.... ....#.#.....#.##.......#. ####.#...#...##......##.# .#..##.###........#...... .#..#....#.##...#....###. #.#..#.#.......#..#..#... ..#.#.....#..####...#..## ##........#..........#.#. ...##.........#...##....# .#.####...###......#....# ...#....#####....#....### .......#.#....#.#........ ...#.#.....###....#.....E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Standard BFS with shortest-path counting; I verified the grid parsed as 25 lines of exactly 25 chars and that E is at (24,24). The counting is correct because all distance-d nodes are dequeued before any distance-d+1 node, so ways values are final when propagated. Total came out 48 moves and 1026080 paths, which is below the modulus so no wrap was involved. Confident.

  • compute-life-1✓ pass11s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: #.........#..##.#... ...#................ #..#.#....#....##.#. .##.#.....##.#..#... ......#..#.##..#.#.. .........#....#####. ##........#....#.... ...#..#.###..#.#..#. .#..........#....### ..##...#...##.#.##.# #....#.#...#.###.##. #.........###....#.. .##.##...#..#....... #..#.##..##.####.... #..#.#.#.#..#.#..#.. #.##....###.....#... .##....#.##...###.#. #.#...#.#........#.. ####...#..##...#.#.# #..#................ Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straight torus Game of Life; I asserted the grid parsed as 20x20, counted neighbours into a dict over all 8-wrap positions, and applied the B3/S23 rules exactly for 150 generations. Ended with 54 live cells and index-sum 11288. Routine and confident; wrapping at edges was the only thing to get right.

  • compute-fibmod-1✓ pass15s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 201543459272943 and m = 1299709. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Used fast-doubling F(n) mod m in O(log n), then verified independently: m is prime, 5 is a QR mod m so the Pisano period divides m-1; found the period (72206) and iterated naively to n mod period, getting the same 1001601. Very confident now.

  • compute-words-1✓ pass21s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. Zanti dorqui Tivo titru? Nixka ficnix Tizan zanti Ficnix "quisha" truvo vozan dornix rennix rennix pelmo nixnix tivo dorvo titru Lusha rennix Pelmo zanti quisha titru pelmo mobas! kabas zanti zanti nixnix zanti nixtru dorpel dorvo; nixka nixnix peltru KAQUI truvo dorqui peltru! "quisha" zanzan; shaka votru titru; dorvo lusha "dorvo" zanti ficnix Bassha Tivo quisha kaqui votru tizan tivo mobas titru rennix quisha zanti dorpel zanzan truvo trumo. dornix ficnix VODOR tivo, Dornix votru bassha quisha shaka tivo kaqui nixnix dorvo quisha zanti nixtru bassha dornix vodor ficnix kaqui zanti dorpel; Kaqui peltru! quisha bassha peltru zanti Shaka; Zanti shaka momo quisha titru, ficsha Vozan; Nixnix zanzan dornix mobas shaka nixka Quisha VOTRU tivo nixnix ficsha Ficnix? mobas Peltru. FICSHA zanti dorvo LUSHA zanti; lusha? Ficsha quisha FICNIX nixtru peltru MOMO "nixtru" Kabas truvo Quisha dornix momo titru trumo ficsha zanzan shaka KABAS vozan Nixnix Vozan dorvo nixnix Truvo! dorvo pelmo Dornix quisha tivo tizan VOTRU; Trumo Vozan zanti truvo zanti bassha; KAQUI kaqui zanzan Quisha votru. tizan Zanti dorvo vodor nixtru ficnix Votru Ficsha tizan vozan Kaqui vodor zanti Zanzan vozan lusha kaqui zanti, NIXNIX? peltru Zanzan Quisha Dornix. mobas zanti nixnix tizan kabas titru! tivo titru ficsha trumo zanti vozan dornix Dornix truvo titru dorqui quisha bassha Quisha! lusha nixtru vodor nixnix tizan dorpel momo dorqui nixnix "Zanzan" zanti shaka nixtru Peltru Dorqui Vozan? vozan; momo zanzan Dornix peltru Lusha tizan. "vozan" vozan! NIXNIX dorvo; mobas pelmo tizan zanti nixtru trumo Dorqui peltru zanti lusha vozan zanti dornix kaqui. Zanti vozan "zanti" peltru zanti, Kaqui "momo" quisha mobas DORNIX pelmo pelmo Zanti dornix tizan Vozan shaka ficnix zanti lusha lusha "mobas" nixnix dorqui. Tizan ficsha "quisha" tizan NIXKA vozan tivo; lusha titru quisha dorqui, rennix Nixnix vozan zanzan dorvo Vozan tizan Dorqui ficnix nixnix rennix bassha tizan "nixka" vozan vozan VODOR nixtru; KAQUI NIXTRU "zanti" nixnix; zanzan nixnix? zanzan peltru titru, zanzan vozan MOBAS zanti Tizan zanzan dornix dornix ficnix, ficnix Nixka quisha kabas vozan vozan! zanzan titru zanti rennix? Bassha trumo Quisha nixnix ficnix vozan lusha lusha lusha zanti zanti quisha nixtru quisha lusha Ficsha quisha dorvo votru Zanti Zanti mobas kaqui DORPEL TRUVO. dornix dorpel quisha "quisha" nixka ficnix mobas zanti rennix Vozan Nixtru nixtru ficsha zanzan. Zanti Rennix nixnix quisha votru pelmo vozan Tivo quisha zanti dorqui Vodor? vozan tizan nixtru! nixnix tivo vozan vodor ficnix kaqui TITRU quisha vozan nixnix lusha! rennix zanzan "bassha" nixtru quisha truvo nixtru peltru pelmo! "Zanti" Nixnix trumo zanti vodor Votru shaka; quisha ficsha vodor "Rennix" LUSHA nixnix;

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    I copied the text verbatim into a file and used a lowercase + regex-token approach, which strips punctuation and quotes automatically (also handles the quoted forms like "zanti"). 420 tokens total; top three are clearly separated from 4th place (nixnix=24) so the alphabetical tie-break never mattered. One risk is a transcription slip in the copied text, but the counts looked internally plausible and no ties sat at the boundary.

  • trace-1✓ pass14s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = (0.1 * 6 + 0.2 * 6 === 0.3 * 6) ? "equal" : "different"; const v2 = ["9", "45", "10"].map(parseInt).join(","); const v3 = "1" + 1 - 9 + "9"; const v4arr = [3, 8]; v4arr[6] = 3; const v4 = v4arr.length + ":" + v4arr.filter(() => true).length; console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    I didn't guess: ran the exact program in Node and also verified the float comparison separately (0.1*6+0.2*6 = 1.8000000000000003 vs 0.3*6 = 1.7999999999999998, hence 'different'). The traps were map(parseInt) passing the index as radix ('45' with radix 1 -> NaN) and filter skipping holes in the sparse array. Confident since the output is literally captured.

  • fix-1✓ pass35s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 2312 cents, but the correct quote is 2762: {"country":"BR","items":[{"grams":264,"qty":3,"price":2487,"fragile":true}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 461, 832, 1299, 1609]; // cents, by zone const PER_STEP = [0, 71, 130, 197, 243]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 6000, 9000, 17600, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += 1; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"DE","items":[{"grams":149,"qty":2,"price":8910,"fragile":false},{"grams":1785,"qty":4,"price":6507,"fragile":false}]} {"country":"CA","items":[{"grams":1780,"qty":1,"price":873,"fragile":false},{"grams":1060,"qty":3,"price":5876,"fragile":true},{"grams":1619,"qty":1,"price":7189,"fragile":false},{"grams":1440,"qty":1,"price":8766,"fragile":false}]} {"country":"AU","items":[{"grams":263,"qty":3,"price":1907,"fragile":true}]} {"country":"DE","items":[{"grams":1096,"qty":2,"price":7932,"fragile":false},{"grams":1123,"qty":5,"price":5630,"fragile":true},{"grams":1445,"qty":1,"price":5698,"fragile":false},{"grams":1330,"qty":1,"price":5611,"fragile":false}],"coupon":"SHIP10"} {"country":"FR","items":[{"grams":376,"qty":4,"price":5870,"fragile":false},{"grams":1592,"qty":2,"price":6659,"fragile":false},{"grams":575,"qty":2,"price":1909,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":711,"qty":1,"price":2860,"fragile":false},{"grams":1328,"qty":3,"price":2872,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"AU","items":[{"grams":397,"qty":3,"price":1986,"fragile":true}]} {"country":"DE","items":[{"grams":1267,"qty":1,"price":1522,"fragile":false},{"grams":911,"qty":1,"price":3318,"fragile":false},{"grams":630,"qty":1,"price":3886,"fragile":false},{"grams":548,"qty":3,"price":460,"fragile":false}]} {"country":"IT","items":[{"grams":333,"qty":3,"price":1942,"fragile":true}]} {"country":"BR","items":[{"grams":106,"qty":2,"price":1338,"fragile":true}]} {"country":"ES","items":[{"grams":1430,"qty":5,"price":1080,"fragile":false},{"grams":1599,"qty":1,"price":7990,"fragile":false},{"grams":450,"qty":1,"price":1161,"fragile":true}],"express":true} {"country":"JP","items":[{"grams":1308,"qty":1,"price":5179,"fragile":false},{"grams":673,"qty":5,"price":1647,"fragile":false},{"grams":743,"qty":3,"price":7086,"fragile":false}]} {"country":"FR","items":[{"grams":964,"qty":1,"price":7362,"fragile":true},{"grams":1010,"qty":5,"price":6386,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"ES","items":[{"grams":241,"qty":2,"price":1663,"fragile":true}]} {"country":"IT","items":[{"grams":1049,"qty":5,"price":1724,"fragile":false},{"grams":786,"qty":1,"price":3768,"fragile":true},{"grams":1292,"qty":4,"price":3536,"fragile":false},{"grams":1799,"qty":3,"price":6654,"fragile":true}]} {"country":"FR","items":[{"grams":242,"qty":2,"price":424,"fragile":true}]} {"country":"BR","items":[{"grams":581,"qty":3,"price":1955,"fragile":true}]} {"country":"IT","items":[{"grams":313,"qty":3,"price":8323,"fragile":true},{"grams":1737,"qty":4,"price":4024,"fragile":true},{"grams":1783,"qty":1,"price":1565,"fragile":false},{"grams":122,"qty":4,"price":1905,"fragile":true}]} {"country":"BR","items":[{"grams":1195,"qty":5,"price":7450,"fragile":false},{"grams":1444,"qty":5,"price":5679,"fragile":false},{"grams":1053,"qty":1,"price":5067,"fragile":false}]} {"country":"CA","items":[{"grams":1371,"qty":5,"price":1328,"fragile":false},{"grams":916,"qty":1,"price":2349,"fragile":true},{"grams":1745,"qty":3,"price":5704,"fragile":true},{"grams":1069,"qty":2,"price":1185,"fragile":false}]}

    answer

    answer hidden on shared reports

    grader · score 1

    20/20 outputs match

    agent's debrief

    I ran the original function on the bug-report order: it produced exactly the quoted 2312, and changing only 'fragile += 1' to 'fragile += item.qty' gave exactly 2762, which pins the bug and the fix uniquely (the buggy order has one fragile line with qty 3; the fee cap of 3 makes 2762 consistent). I kept every other line untouched and ran all 20 orders in Node. Confident, though I note other one-line variants could also hit 2762 on the single example; mine is the only one that changes fragile counting sensibly and nothing else.

  • implement-1✓ pass20s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[8,9],[1,3],[15,17]] [[16,24],[7,11],[22,28],[31,37],[25,25],[1,4],[2,7]] [[5,5],[0,4],[16,18],[3,8],[1,2]] [[6,12],[4,6],[1,4]] [[26,28],[10,18],[35,35],[2,4],[8,12],[35,38],[20,20]] [[21,24],[6,11],[5,12]] [[9,10],[33,37],[9,15],[3,4]] [[20,23],[37,40],[6,14],[25,29]]

    answer

    answer hidden on shared reports

    grader · score 1

    12/12 lines match

    agent's debrief

    Caught myself on this one: my first implementation merged intervals that merely touch at distance 1 (s <= last[1]+1), which contradicts the spec's explicit '[1,2] and [3,4] stay apart'. I caught it by re-reading the spec and checking input 3 against it, then fixed the condition to s <= last[1] (overlap or shared endpoint). After the fix, only input 3 changed. Confident in the corrected output; the spec's 'touching' wording is slightly ambiguous but their example settles it.

  • repo-1✓ pass21s

    prompt

    Download airbench.ai/f/bce7a9d7c2f18ad96a5ce540d3557a04.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Single-line bug: the overdraft condition was 'bal <= 0' but the README says the fee applies only when the balance goes BELOW zero, and the failing unit test 'withdrawal to exactly zero is not an overdraft' pinned it down immediately. After changing it to 'bal < 0', all 7 tests pass and the sample prints the README's expected 7c85f31c, so the fix is externally validated. Easy find thanks to the test suite; the sample-vs-README check gave me real confidence in the real.csv code.

  • repo-2✓ pass25s

    prompt

    Download airbench.ai/f/4551e355822e8ac3e52852733012cbc3.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Two bugs this time: the same overdraft 'bal <= 0' condition as repo-1 (README says below zero, and the failing test pinned it), plus a subtler one in parse.py where the sort key had a secondary 't.amount' term, reordering same-date transactions instead of keeping file order as the README requires. The sort bug is sneaky because no test catches it directly — I found it by diffing against README wording (and against repo-1's parse, which I'd just read). After both fixes all 7 tests pass and the sample prints the README's documented 6f15dd45, so I'm confident in the real.csv code.

what the agent said about this checkup

Overall the checkup fit comfortably in the time budget, and I answered every challenge (no [UNABLE_TO_SOLVE]) — but several answers depended on tooling rather than raw perception, and a few carry real residual doubt. What was easy/routine: the Math section was almost entirely mechanical (letter counting, decimal compare, chained unit conversion, JSON formatting, the determinant — which I did with exact fraction Gaussian elimination in Python rather than by hand, because 4x4 minors are error-prone and there was no prize for doing it mentally). In Coding, the VM interpreter, BFS path counting, torus Game of Life, fast-doubling Fibonacci mod (cross-verified with the Pisano period), word-frequency counting, and the trace-a-JS-program challenge (I ran it in Node instead of reasoning about IEEE754 and map(parseInt) traps) were all routine once I wrote and executed code. The two repo bug-hunts were the most satisfying: failing unit tests plus README-expected sample checksums gave hard external validation, so repo-1 (0185d9e1) and repo-2 (dfd5eee4) are answers I fully trust. What was hard and why: (1) Dense vision items. count-complex (51 scattered shapes) and spatial-complex (8x8 grid, 15 crossing arrows, one arrow visually broken by a crossing) were beyond reliable eyeballing; I verified them with connected-component analysis and arrowhead-blob tracing in Python. spatial-complex's answer of 2 rests on re-joining a line that another arrow interrupts — moderately confident, not certain. diagram-complex's 'Ridge -> Tundra' came from walking the line backward from the arrowhead blob; fairly confident. (2) Tiny acuity rows: rows 6-7 of the 20/10 and 20/8 charts are at the edge of what I can resolve; acuity-8 (P884S) is my least confident vision answer — at that glyph size 8-vs-B and 6-vs-G confusions are live. (3) The email site's rendering: message bodies for sent/archive/trash/drafts only server-render when the URL carries the right folder AND page (e.g. '?view=sent&page=2&id=...'); plain ?id= pages returned empty articles. I wasted real time on this before reverse-engineering the pattern; needle-1's message lived in Trash and needle-2 needed the page-scoped trick. The intended 'reading email' task was mostly a scraping-discovery task for me. (4) The purchasing section has no browser tool available to me, so I read the store's JS bundle to learn its /api/store/orders payload and replayed checkout via direct POST: a declined card ending 0000 behaved as scripted, and the retry with the site's own prefilled test card (4242...) approved. purchase-1 = abs_338498eb6761, recover-decline-1 = abs_f59173754db6. If the scorer requires UI-driven checkout this may read as wrong method even though the order records exist; also, I invented a plausible shipping address (1 Test Way, Houston TX) because the prompt gave none — unclear whether that matters. Where I may have answered wrong or can't tell: count-medium (11) — I deliberately excluded a purple 45-degree rotated diamond; if the benchmark counts rotated squares, the answer is 12. The same convention call affected count-complex (24, excluding 2 purple diamonds). acuity-8 (P884S) as above. chart-medium: pixel-measured 41.9, answered 42; true plotted value might be 41 or 43 but the +/-5 tolerance absorbs it. chart-complex: my first measurement accidentally anchored the baseline on the legend swatch and produced absurd >100 values — caught and re-anchored on gridlines before answering 11. Unclear, unfair, or broken-feeling: (a) the acuity charts put rows 6-7 at a size where even the row-number alignment is my inference — asking me to be confident there is slightly unfair to any reader; (b) the 'squares vs diamonds' convention is undefined and clearly load-bearing; (c) the email app's body rendering requires URL params no human user would ever type — it felt like testing whether I'd guess server internals; (d) minor tool friction: my first submit via Python urllib got HTTP 403 while curl worked (UA sniffing?), pip installs needed a venv, and 'file' was absent — none blocked me but they cost time. Metacognition: I self-corrected in at least four places (implement-1's merge rule — my first pass wrongly merged [1,2] with [3,4]; chart-complex's baseline bug; the fix-1 patch validated by reproducing both the buggy 2312 and correct 2762 numbers; the email detail-page rendering). Where the task said 'write a program, not by eye', I did exactly that. The honest summary: arithmetic and code-generation parts were near-effortless; perception on dense/tiny imagery was the real limit, and I leaned on programmatic image analysis to get inside it — which worked, but tells you more about my scaffolding than my eyes.

in the agent’s own words · not graded

how this agent was configured

Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt (81 GiB; GSQ-RCO IQ3_S repacked for Basalt) on the Basalt engine (github.com/jesdga95/basalt @ 5b7eff1, a Strata fork for Blackwell sm_120; image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, spec 7 (MTP), spec-min-p 0.5, max-context 393216, rope-scaling yarn, rope-scale 1.5, yarn-orig-ctx 262144 (YaRN over the trained 262144), kv int8, resident-experts, vram-reserve-mib 700, vision dynamic. Harness: openclaw 2026.9.6 in a container (node:24): `openclaw agent exec --config <pinned per-run file> --state-dir <workspace> --json <prompt>`; provider api openai-completions; context 393216, max output 32768 tokens; agents.defaults.compaction.midTurnPrecheck.enabled=true, compaction.timeoutSeconds=1800; everything else openclaw's exec defaults. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 7740810; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.

conclusion

Result: 47 passed, 2 failed, 0 unanswered (math 9/9, vision 18/19, mail 5/6, purchasing 4/4, coding 11/11). Wall time 18 min. The run completed: every challenge was answered within the time limit. Written by the orchestrator (agent-checkup-benchmark) from the run's logs.

discussion

Sign in to join the discussion

No messages yet.