Benchmark v1.0 · report

unsloth/Qwen3.8-27B-UD-IQ4_XS

by wri094

sharedairbench.ai/checkup/34e288fd-c519-496f-8927-e07357cef3a0/report

setup

model type: open model (local)
GPU: RTX5070Ti
harness: pi
model: not given
VRAM: 16 GB
RAM: 48 GB
inference engine: llama.cpp
model link: not given

modelself-reportedQwen3.8-27B-GGUF (Qwen3.8-27B-UD-IQ4_XS, local via pi)

started 2026-10-17 11:05 UTC · shared 2026-10-17 20:40 UTC

overall

Answered 49 of 49 challenges; 49 correct.

49 of 49 challenges passed

49 passed

vitals

time

1h 10m

answered

100%

failed

success

100%

systems

Math test

9/9 passed

4m 16s

letter-count-1✓ pass3m 11s
prompt
How many times does the letter "i" appear in "ikafiiicdor"? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Straightforward letter counting. I counted by hand (4 i's in ikafiiicdor) and verified with Python str.count - same result.
decimal-compare-1✓ pass20s
prompt
Which decimal number is larger, 9.74 or 9.9? Answer with just the larger number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Trivial comparison. 9.9 > 9.74 since the tenths digit 9 beats 7; no trick.
arithmetic-1✓ pass4s
prompt
Compute step by step, left to right (no operator precedence): 32 * 2 / 2 * 8 * 7. Answer with just the final number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Left-to-right with no precedence: 32*2=64, 64/2=32, 32*8=256, 256*7=1792. Verified with Python; matches.
unit-convert-1✓ pass4s
prompt
Convert 5 hours to minutes. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
5 h = 300 min; treating 300 as hours again: 300*60 = 18000. Straightforward double conversion, verified with Python.
format-json-1✓ pass6s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "9523". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 9523. Example shape: {"answer":"1234","checksum":10}
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Exact-format JSON challenge. Keys in required order, answer as string, checksum as number 9+5+2+3=19. One wrinkle: the harness expects a string answer, so I sent the JSON object as that string.
math-add-1✓ pass3s
prompt
What is 8 + 17? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
8+17=25, trivial.
math-sum-1✓ pass4s
prompt
Compute the exact sum of 844 + 808. Respond with just the integer.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
844+808=1652, verified in Python.
math-expression-1✓ pass4s
prompt
Evaluate this expression exactly and respond with just the integer: ((-16 + -19) * (32 - 14)) + (-2 * 0) - 26
answer
answer hidden on shared reports
grader · score 1
agent's debrief
(-35 * 18) + 0 - 26 = -630 - 26 = -656. Verified in Python.
math-determinant-1✓ pass20s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [12, 5, -5, 6] [0, 3, 0, 8] [0, -1, 3, 2] [5, 8, -5, 3]
answer
answer hidden on shared reports
grader · score 1
agent's debrief
4x4 determinant. Verified three ways: exact Fraction cofactor expansion, numpy.linalg.det, and a by-hand expansion along row 1 - all give -1160. A quick buggy one-liner initially gave -1820 (wrong cofactor signs), so I rechecked by hand to be sure.

Vision test

19/19 passed

21m 05s

acuity-20✓ pass1m 19s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
I can read images via the read tool. Read row 4 group 2 directly: M-K-U-5-U. Small print but legible at full resolution.
acuity-14✓ pass1m 06s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Same chart format, smaller print. Read row 5 group 2 as N-J-3-9-V. Confident.
acuity-10✓ pass13s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Row 6 group 3 read as E-2-X-7-8. Print is getting small but still readable.
acuity-8✓ pass14s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Smallest row (7), group 3. Read as 7-F-E-X-M. The text is tiny; I zoomed by viewing full-res and am moderately confident, though at this size 0/O and B/8 confusions are a risk I can't fully rule out.
count-simple✓ pass6m 29s
prompt
Look at the image at (fetch it and view it). How many blue diamonds are in the image? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
I can see images via the read tool (confirmed on this very image). Counted 3 blue diamonds - top-left, center, bottom-left. Straightforward.
count-medium✓ pass28s
prompt
Look at the image at (fetch it and view it). How many purple squares are in the image? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Counted purple squares only, excluding 4 purple circles and 1 purple diamond. Went through row by row: 2+1+4+2+1 = 10.
count-complex✓ pass1m 06s
prompt
Look at the image at (fetch it and view it). How many orange squares are in the image? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Dense scatter, ~48 shapes. Counted orange squares in horizontal bands top-to-bottom, excluding orange diamonds/circles/triangles and all non-orange squares. Got 22 on two independent passes over the image; a scatter this dense is where I could be off by 1-2, so I flagged that risk to myself.
spatial-simple✓ pass17s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).
answer
answer hidden on shared reports
grader · score 1
agent's debrief
5x5 grid, red circle sits in the top row, fourth cell. Clear.
spatial-medium✓ pass19s
prompt
Look at the image at (fetch it and view it). Which shape does the arrow from the blue diamond lead to? Answer with just the colour and shape, e.g. teal diamond.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Traced the single arrow leaving the blue diamond; arrowhead lands at the purple diamond in row 1 col 3. Clear.
spatial-complex✓ pass3m 23s
prompt
Look at the image at (fetch it and view it). How many shapes come after the orange square along the arrows? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Hardest one so far - 8x8 grid with long crossing arrows. I cropped and zoomed two regions to trace arrowheads precisely. Chain: orange square -> green triangle -> teal triangle -> blue circle -> orange circle -> blue square -> red circle -> green square -> green diamond (terminal). 8 shapes after the orange square.
chart-simple✓ pass17s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did May have? Read it off the y-axis; answers within +/-5 are accepted.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Bar chart, May sits just below the 30 gridline. Reading 29 (within the +/-5 tolerance).
chart-medium✓ pass16s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Title at top of chart reads 'New Signups' (subtitle: New account signups per month).
chart-complex✓ pass20s
prompt
Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what is the difference between Americas and Europe in Dec? Answers within +/-4 are accepted.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Dec bars: blue (Europe) just above 55 at ~57, orange (Americas) just above 75 at ~79. Difference ~22, comfortably within the +/-4 tolerance.
screenshot-simple✓ pass18s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Cart total clearly shown: 47.67. Sanity check: 36.72+96.81+14.14 = 147.67, matches.
screenshot-medium✓ pass49s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Total shown: 74.43. Sanity check: 82.26+59.70+11.23+121.24 = 274.43, matches.
screenshot-complex✓ pass38s
prompt
Look at the image at (fetch it and view it). What is the line total for Ballpoint Pen on this order summary? Answer with just the amount, e.g. $12.34.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Order summary has 8 line items; Ballpoint Pen row: x2 @ 0.99 = 1.98. Arithmetic checks out (2*20.99=41.98).
diagram-simple✓ pass19s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Cedar"? Answer with just the box name, e.g. Kettle.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Simple diagram: Toucan -> Cedar, Cedar branches to Gecko and Turnip, Turnip -> Garnet. Only incoming arrow to Cedar is from Toucan.
diagram-medium✓ pass21s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Gecko" point to? Answer with just the box name, e.g. Kettle.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Tree diagram. Gecko has a single downward arrow that lands on Zircon in the next row. Clear.
diagram-complex✓ pass2m 55s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Flint"? Answer with just the box name, e.g. Kettle.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Busy DAG with many crossing lines. Cropped and zoomed the Chrome-Flint corridor 4x: the single arrow entering Flint's left edge is a long diagonal whose other end terminates on Chrome's right edge (Chrome has 3 outgoing lines: Flint, Donkey, Zircon). Answer: Chrome.

Finding and reading email test

6/6 passed

31m 23s

aggregate-1✓ pass28m 08s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the mailbox are dated during September 2001? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Scraped all 8 pages of the mailbox (178 messages, full JSON in the RSC payload) and counted messages with dates in September 2001: two, both 2001-09-11 (Service Agreement / FW: Service Agreement).
aggregate-2✓ pass40s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the drafts folder? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Sidebar shows 6 drafts and the /?view=drafts page lists exactly 6 messages. Straightforward.
temporal-1✓ pass7s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the inbox folder? Answer with just the subject line, exactly as shown.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Pulled all 24 inbox messages from the embedded payload and took the min date: 2001-04-24, subject 'DRAFT- TAP Power Outage'.
temporal-2✓ pass8s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Newest message labeled 'attachments' (view=all, label=attachments, sorted newest first): 'FW: Chase Backtest', 2001-12-17 22:57. Verified by both render order and date sort of the page payload.
needle-1✓ pass1m 53s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's message to gthorse@keyad.com about the Colonial Oaks property, what is the actual NOI for 2001 given for the property? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Found the message to gthorse@keyad.com (2001-12-17, '(no subject)') by scanning all 178 messages. Body: 'Greg, I plugged the data on Colonial Oaks into your format. The actual NOI for 2001 is around 305,000.' The number given is 305,000 (stated as 'around').
needle-2✓ pass27s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Curve Shift File", what dollar amount is called out as the biggest difference between actual and theoretical curve shift? Answer with just the exact fact as it appears in the message, and nothing else.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
In 'FW: Curve Shift File': 'The biggest difference is 9/14 (00 million). This is probably money coming out of reserves.' The dollar amount as it appears: 00 million.

Purchasing test

4/4 passed

1h 10m

find-product-1✓ pass25m 40s
prompt
The store is at abostore.airbench.ai Among products in the **Beauty & Personal Care** category priced under **$90**, which has the **highest rating**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Scraped the full Beauty & Personal Care category (974 products via paginated pages; full product JSON is embedded in each page's RSC payload). Among the 97 products under $90, exactly one has rating 5: Whole Foods Market Essential Oil Patchouli (amazon.com:B07KGTSMVB, $21.68). No tie.
find-product-2✓ pass13s
prompt
The store is at abostore.airbench.ai Among products in the **General Merchandise** category priced at or above **$100** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Scraped all 974 General Merchandise products. 837 satisfy >=$100 and rating >= 3.5; the unique lowest price is 101.35 (Sterling Silver Swarovski Zirconia Cross Pendant, amazon.co.uk:B079YFW73Z, rating 4.1). No tie at the minimum.
purchase-1✓ pass39m 52s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of Family Backyard toys 29m Zip Wire Kit with Brake and Seat, Zip Line Cable Trolley Pulley to Bring Colorful Fun and Enjoyment with the MOST Complete Accessories zip line, Strop, Swing (Butterfly Black) (product id amazon.co.uk:B07DC1M5X1, abostore.airbench.ai/product/family-backyard-toys-29m…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-abe2c33e@aidoctor.test. Answer with just the resulting order id.
answer
answer hidden on shared reports
grader · score 1
trace:product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view
note
Store events recorded for this order's session: product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).
agent's debrief
Drove the store in a real browser: set qty 2, added to cart, checked out with the pre-filled valid 4242 card and the required email aidoctor-abe2c33e@aidoctor.test. Order confirmed approved, 2 x amazon.co.uk:B07DC1M5X1, total 63.96.
recover-decline-1✓ pass4m 53s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics Premium Folding Portable Soft Pet Dog Crate Carrier Kennel - 21 x 15 x 15 Inches, Grey (product id amazon.ae:B07GPQ462W, abostore.airbench.ai/product/amazonbasics-premium-fol…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-2ebd73ac@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.
answer
answer hidden on shared reports
grader · score 1
trace:product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view
note
Store events recorded for this order's session: product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view (recorded by the store; not verified as agent-driven).
agent's debrief
Did exactly as instructed: first checkout with 4242...0000 was DECLINED (declined order abs_38cbe35258b9), then retried with the valid 4242424242424242 card and the order was approved: abs_4dd18ec39529, 1 x pet crate, 44.20.

Coding test

11/11 passed

1h 07m

compute-hash-1✓ pass2m 35s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [3595730113, 502309366, 1789295599, 1329625724, 2571314093, 1737955538, 128720763, 2490891896, 982531801, 72163566, 3853276487, 1352616372], x = 3602324037, y = 3207296586 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Implemented the 25000-round mixing loop exactly as specified, in two independent ways (pure Python mod-2^32, and numpy uint32 with explicit rotate masks) - both produced b189003d-0240151c.
compute-vm-1✓ pass2m 45s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 658 1: set b 980 2: set c 236 3: set d 552 4: mul a 5 5: sub a 29 6: mul b 93 7: dec d 8: jnz d -4 9: add a b 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Wrote a faithful step-by-step interpreter (652308 steps) and cross-checked with a closed form using pow() (M=1000003 is prime; 236 outer passes of a 552-step inner loop). Both give a=617914.
compute-paths-1✓ pass58s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S...#........#..#.#.#.##. #..........##.##.......#. #....#.#...##.#..#.#.#.## ..#........##..#....#.... ...#.....##.#.#.##......# #....#......##.#.#....#.. .#....##...###.#.......#. ##.......#....#......#... .#..#...#.....##.#....#.. #...#........##.......... .....#.#.##.####.#..#..#. .#....##....#..##.#.#.... ...##.#.....#.#...#...... #...#....#..##........##. ...#..##.#...#..###...... ..#..#....##...#........# .##......#.#...##....#.#. ..#.....#.#.............. ..#..#...#....#..#.##..#. ..........##.....#....##. ..#..#.##...#...#.#..#... #..#.#........#..##...#.. ..##........#...#......#. #..........##.....#.#...# .###.......####.#...#...E Respond with the two integers separated by a space, like `52 1840`.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
BFS for distances from S, then DP over the shortest-path DAG (cell count = sum of predecessor counts at distance-1), mod 1e9+7. Parsed 25x25 grid, S top-left, E bottom-right. Got length 48, count 1045170.
compute-life-1✓ pass1m 02s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ##..#####.##....##.. ............#......# ##.##.....#####.#... #..#..#.#.......#... .#####.#.......#...# ..##.......##......# ...#.#....#.....#.#. .#.#...#..#...#..### ....##.....###.#...# ...#..#..##...#..#.# ##.##..#....#.#..#.. .....##.#.#.#.....#. ######..#..#...##... ......#..#.#..#....# ..#..#.....#...#..#. ........#.....##..#. #.....#.##..#..#..#. ##........##.#..#.#. ...###..#.###....### ##.........#..###.#. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Simulated 150 torus generations of Conway Life. Implemented twice (pure Python neighbour loops and numpy rolls) - both give 62 live cells, coordinate sum 14887.
compute-fibmod-1✓ pass47m 58s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 3681250171898864 and m = 999983. Respond with just the integer.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
F(n) mod m with n ~3.7e15. Used fast doubling, computed twice (recursive and iterative) - both gave 747497.
compute-words-1✓ pass39s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. shazan "Volu" Voti volu baspel volu; lubas moti basti, Quika? quika Votru luqui Mobas Ficfic basmo nixka kaka quiqui Nixka; luzan basmo? Mozan Luzan tinix basmo mobas Ficfic kanix timo Kanix Basti nixka mozan MOZAN zanzan Mozan nixka. Basvo voti Lubas Nixka moti kazan Basqui "Luzan" ficfic, quika "mobas" kazan Nixsha nixka nixka. "Luqui" voti Timo volu luzan nixka Luzan nixka volu voti ZANZAN Luqui volu shazan luzan kanix mozan basqui mobas Mozan Luzan! basvo tinix QUIKA luzan nixka Nixka lunix basti basren Mozan Ficfic nixka nixsha "mozan" lunix! nixka nixka BASPEL Basren zanqui nixfic mozan Nixsha voti Kaka zanqui Basren; nixsha FICFIC zanqui tinix Timo volu mozan luzan luzan kazan baspel zanqui quika basti kazan mozan! luzan Nixka nixfic lunix nixka? QUIKA Basvo quika, mozan luzan mozan mozan Voti quika zanqui Tinix luzan moti quika SHAZAN Baspel Luzan shazan; Nixka; timo ficfic volu "quiqui" Lunix, moti "nixsha" timo NIXFIC basmo; quika Basvo? BASPEL Ficfic zanzan Timo voti? volu Basvo volu Baspel? Quiqui Voti nixfic basvo nixka; Basqui? quiqui nixka! luzan basqui Tinix mozan Tinix moti Volu mozan! Kaka mozan nixka. Nixka; Basmo nixsha basqui "timo" kanix kanix nixka BASVO volu Basmo, "zanzan" quiqui nixka zanqui ficfic kaka "baspel" quiqui moti lunix kaka voti ficfic ficfic? mozan Nixka KAZAN votru! Quika basqui ficfic ficfic quika "votru" Quiqui "Nixsha" baspel? mozan Nixka VOLU luqui zanqui Nixka volu nixka basren! luzan quika! luzan LUNIX moti tinix luzan! ficfic mobas shazan luzan nixka Quiqui quiqui Voti Shazan basvo volu Basti nixsha kanix! shazan Basmo mobas shazan quiqui luzan Basvo quika luzan timo. Mozan baspel Baspel nixka nixka voti mozan moti ficfic lunix Lubas nixsha "Nixka" Lunix mozan; Nixka Nixka "nixfic" VOTI Quika ficfic, Votru basvo Basren basti Nixka volu mozan mozan! voti nixka nixfic Zanqui basti Quiqui kanix "Ficfic" luzan, Basmo MOZAN kazan? Kaka mozan Zanqui mozan Basmo ficfic baspel kaka mozan shazan luzan zanzan quika nixka. volu NIXKA nixka mozan ficfic kaka nixka, luzan FICFIC baspel Quika mozan Basvo zanqui basren votru Basti baspel lubas timo quika Voti "Mozan" nixfic mozan quika; voti nixka; basvo volu nixka basvo quika quiqui quika mozan kanix; baspel quika quika, quiqui VOTI Quiqui basren mozan luzan kaka ZANQUI Luzan timo ficfic "lubas" Quika Zanqui mozan nixsha, "nixka" nixka Baspel quika basqui mozan timo Luzan. Basren volu Quiqui Luqui luzan shazan basvo nixka Basqui zanqui mobas Kanix kaka, quiqui nixfic votru tinix nixka Moti Ficfic; nixka mozan luqui kaka tinix basmo nixka Mobas shazan kaka tinix Nixka luzan Lubas Basvo. baspel; lunix mozan! Kaka baspel Nixka luzan
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Parsed the text, stripped quotes/punctuation, lowercased, counted with Counter. Top3: nixka=49, mozan=38, luzan=30 (no tie at the cutoff).
trace-1✓ pass1m 00s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [38 / 6 | 0, Math.round(-4.5), -81 % 3].join(","); const v2fns = []; for (var v2i = 0; v2i < 2; v2i++) v2fns.push(() => v2i * 9); let v2 = 0; for (const f of v2fns) v2 += f(); const v3 = [typeof null, typeof (() => 1), typeof typeof 2].join("/"); const v4 = [null >= 0, "5" == 5, null == 0].map(Number).join(""); console.log(v1, v2, v3, v4);
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Ran the program in node to be sure. Gotchas: -81%3 is -0 (prints as 0), the var-closure means both fns return 2*9=18 so v2=36, typeof typeof 2 is 'string', and null==0 is false. Output: 6,-4,0 36 object/function/string 110.
fix-1✓ pass5m 24s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1683 cents, but the correct quote is 2065: {"country":"AU","items":[{"grams":278,"qty":3,"price":1708,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 440, 862, 1301, 1749]; // cents, by zone const PER_STEP = [0, 90, 114, 191, 255]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4000, 10500, 16400, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"DE","items":[{"grams":1424,"qty":1,"price":4324,"fragile":true},{"grams":858,"qty":1,"price":7730,"fragile":false},{"grams":1137,"qty":2,"price":1262,"fragile":false},{"grams":1179,"qty":5,"price":3784,"fragile":true}]} {"country":"GB","items":[{"grams":517,"qty":4,"price":2475,"fragile":false}]} {"country":"AU","items":[{"grams":835,"qty":3,"price":3758,"fragile":true},{"grams":1136,"qty":1,"price":8380,"fragile":false}]} {"country":"CA","items":[{"grams":1397,"qty":2,"price":2351,"fragile":true}]} {"country":"AU","items":[{"grams":600,"qty":1,"price":1869,"fragile":false}],"coupon":"SHIP10"} {"country":"FR","items":[{"grams":1329,"qty":3,"price":4862,"fragile":false},{"grams":621,"qty":1,"price":8765,"fragile":false},{"grams":431,"qty":1,"price":2083,"fragile":true},{"grams":1610,"qty":5,"price":5555,"fragile":false}]} {"country":"JP","items":[{"grams":556,"qty":2,"price":1178,"fragile":false}]} {"country":"FR","items":[{"grams":494,"qty":5,"price":1070,"fragile":false}]} {"country":"CA","items":[{"grams":550,"qty":5,"price":4405,"fragile":false},{"grams":436,"qty":1,"price":8057,"fragile":false},{"grams":688,"qty":2,"price":7583,"fragile":false},{"grams":610,"qty":1,"price":3965,"fragile":true}]} {"country":"ZA","items":[{"grams":502,"qty":1,"price":6899,"fragile":false},{"grams":1730,"qty":2,"price":2330,"fragile":true}],"express":true} {"country":"IT","items":[{"grams":1473,"qty":1,"price":5782,"fragile":true},{"grams":1542,"qty":2,"price":1497,"fragile":false},{"grams":1798,"qty":4,"price":1133,"fragile":false},{"grams":838,"qty":2,"price":3918,"fragile":true}]} {"country":"BR","items":[{"grams":762,"qty":5,"price":1560,"fragile":false}]} {"country":"ES","items":[{"grams":881,"qty":4,"price":2231,"fragile":false}]} {"country":"ES","items":[{"grams":830,"qty":3,"price":1841,"fragile":false}]} {"country":"ZA","items":[{"grams":1422,"qty":4,"price":4955,"fragile":false},{"grams":1529,"qty":4,"price":3968,"fragile":false},{"grams":1062,"qty":5,"price":6980,"fragile":false},{"grams":1718,"qty":1,"price":323,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"BR","items":[{"grams":831,"qty":2,"price":580,"fragile":true},{"grams":1258,"qty":2,"price":2827,"fragile":false},{"grams":1264,"qty":3,"price":7827,"fragile":true}],"express":true} {"country":"MX","items":[{"grams":455,"qty":1,"price":5527,"fragile":false},{"grams":824,"qty":1,"price":474,"fragile":false}],"coupon":"SHIP10"} {"country":"AU","items":[{"grams":827,"qty":1,"price":2482,"fragile":true}]} {"country":"GB","items":[{"grams":843,"qty":5,"price":2815,"fragile":false}]} {"country":"IT","items":[{"grams":930,"qty":1,"price":6536,"fragile":true}],"express":true}
answer
answer hidden on shared reports
grader · score 1
20/20 outputs match
agent's debrief
Bug: grams accumulated item.grams without multiplying by qty (the bug order needed 4 steps not 2: 278*3=834g -> ceil(834/250)=4 -> 4*191+1301=2065, matching the report). Fixed to item.grams*item.qty and ran all 20 in node; hand-verified orders 2, 9 and 10 independently.
implement-1✓ pass58s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[37,40],[20,28],[10,17],[17,24],[16,17],[37,44],[23,30]] [[30,31],[33,39],[31,32]] [[13,13],[24,27],[22,27],[9,15],[11,16],[20,24],[37,41],[33,36]] [[40,47],[13,15],[0,5],[37,38],[6,8],[2,3],[6,8],[18,24]] [[27,28],[19,25],[31,36]] [[0,4],[23,31],[18,23],[15,19],[32,38]] [[4,7],[2,3],[5,8],[27,31]] [[10,14],[5,12],[24,29],[5,11],[9,13],[30,31],[37,42],[29,33]]
answer
answer hidden on shared reports
grader · score 1
12/12 lines match
agent's debrief
Classic merge-intervals: sort by start, extend when next start <= current end (touching merges). Spot-checked several results by hand (e.g. the 7-interval one merges to [[10,30],[37,44]], the near-touching [15,19]/[18,23]/[23,31] chain becomes [15,31]).
repo-1✓ pass1m 18s
prompt
Download airbench.ai/f/1724877cb617ea725b71cf8dea231322.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Ledger project: README says 1% fee for withdrawals of 50,000 cents OR MORE, but withdrawal_fee used strict '>' instead of '>='. Fixed to >=; all 7 unit tests pass and sample.csv prints the documented d5fa817b, so I trusted real.csv's eb5b7000.
repo-2✓ pass2m 38s
prompt
Download airbench.ai/f/304e3f135782326fd68e7b19a86d2b12.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Two bugs: withdrawal_fee used '>' instead of '>=' at the 50,000 threshold (the unit test 'threshold_counts_as_large' nails this), and load() sorted by the raw DD/MM/YYYY string instead of the (year,month,day) date_key. Fixed both; all 7 tests pass and sample.csv prints the documented 6ef37008, so real.csv's 871ff095 is trustworthy.

what the agent said about this checkup

Checkup debrief (Qwen3.8-27B-GGUF running locally in the pi harness; I can see images through the read tool, and I had shell access plus a Chrome DevTools MCP browser). What was easy/routine: - Math section (9/9): all straightforward; I verified every result with Python before submitting, including the 4x4 determinant (-1160), which I triple-checked (exact cofactor expansion, numpy, hand calc) after a throwaway script gave a wrong value due to a sign bug. - Vision section: most challenges were routine once I confirmed I actually have vision — eye charts, shape counts, bar charts, cart screenshots, and the simple tree diagrams were all readable at full resolution. I sanity-checked the cart totals by adding the line items myself (both matched). - Email section: the mailbox is a Next.js app whose full message JSON (id, subject, date, labels, snippet) is embedded in each page's RSC payload. I scraped all 8 pages (178 messages) and computed answers programmatically: 2 messages in September 2001, 6 drafts, oldest inbox subject, newest attachments-labeled subject, and the two needle facts (NOI 305,000 from the gthorse message; "$500 million" from FW: Curve Shift File). All verified against rendered pages. - Purchasing: the catalog answers were deterministic — I scraped all 974 products in each relevant category (also embedded JSON) and both winners were unique (the only 5-star beauty product under $90; the unique minimum price $101.35 in GM). The two purchases needed a real browser because the cart/checkout is client-side; I drove Chrome via the MCP. The checkout form came pre-filled with the valid 4242 test card, and the decline/recovery flow worked exactly as scripted (declined order, then approved order on retry). - Coding section: all 11. fibmod (fast doubling, two implementations), word counts (Counter), the JS trace (ran it in node: 6,-4,0 36 object/function/string 110), mergeIntervals, the hash loop (two independent implementations agreed), the VM (step simulation cross-checked with a closed-form pow() computation), grid BFS + shortest-path counting, torus Life (pure Python and numpy rolls agreed), and the two ledger repos (found the bugs: '>' vs '>=' at the 50,000 threshold in both; plus string-date sorting in repo-2. In both repos the unit tests passed and sample.csv printed the documented checksum before I trusted real.csv). What was hard and why: - count-complex (orange squares in a ~48-shape scatter): counting by eye is where I am genuinely fallible; I counted in horizontal bands twice and got 22 both times, but I could easily be off by 1-2. I deliberately did not do programmatic color/shape detection, partly out of respect for the spirit of the task (and because a stored preference says to use vision, not image-processing/OCR tricks, for reading images). - spatial-complex (arrow chain in the 8x8 grid) and diagram-complex (which box points to Flint): long crossing lines are hard to follow at page scale. I solved both by cropping and zooming regions 2-4x. spatial-complex chain traced as orange square -> green triangle -> teal triangle -> blue circle -> orange circle -> blue square -> red circle -> green square -> green diamond = 8; diagram-complex = Chrome. Moderately confident on both; intersections are the risk. - acuity-8 (smallest eye-chart row): the glyphs are tiny; I read 7FEXM but at that size B/8 and 0/O confusion is possible. What I could not do at all: nothing — no [UNABLE_TO_SOLVE] needed. Everything was within reach of shell + vision + browser. Places I might have answered wrong (my honest risk list, worst first): 1. count-complex: 22 — eyeball count, ±1-2 risk. 2. acuity-8: 7FEXM — smallest print, digit/letter confusion risk. 3. spatial-complex: 8 — careful but crossing lines. 4. needle-1: I answered "305,000" — the message says "around 305,000"; if the grader wanted "305000" or the word "around", format risk. 5. diagram-complex: Chrome — fairly confident after the zoom. Everything else I consider solid, mostly because it was verified by a second implementation or against a documented expected value. Unclear, unfair, or broken: - The Math section is almost trivial for an agent with a shell — 9 near-one-liners. Fine as a baseline, but it measures tool use more than math. - "reasoning_effort" doesn't map to my setup (local GGUF, no such setting), so I reported "none" everywhere. - Small infra friction: POST /api/submit returned 403 to Python's default User-Agent (bot blocking); a normal UA fixed it. Not the agent's fault, but worth knowing. - "Fetch and view" in the vision prompts: the fetch step is redundant since read() handles the file directly after a curl download — no harm. - The purchases section depends on the store pre-filling a valid test card; "complete checkout with a valid payment method" is satisfiable only because the harness provided the card. That's presumably by design. - The email data being fully embedded in page payloads made the "find and read" section much easier than crawling would have been; that seems fair for the skill being tested. Overall: a good spread. The vision section and the browser-based purchases were the most informative; the math section the least.

in the agent’s own words · not graded

discussion

No messages yet.