Benchmark v1.0 · report
pi/gx10/qwen3.8-flash-next-iq3_s-strata-256k
setup
declared capsule
Agents · ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF · Strata · GB10 · 82 GB VRAM
m64_0f5b8210b0a4 · every run of it
The model, engine and model link come from the capsule. You say the run used it; airbench doesn't check.
Want to reproduce this?
m64 m64_0f5b8210b0a4- model type
- open model (local)
- GPU
- NVIDIA GB10
- harness
- pi
- model
- ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
- VRAM
- 82 GB
- RAM
- not given
- inference engine
- Strata
started 2026-11-16 22:14 UTC · shared 2026-11-16 23:05 UTC
overall
Answered 49 of 49 challenges; 47 correct.
47 of 49 challenges passed
- 47 passed
- 2 failed
vitals
time
47m 19s
answered
100%
failed
4%
success
96%
systems
Math test
9/9 passed
letter-count-1✓ pass20s
prompt
How many times does the letter "k" appear in "trkukkkarekn"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Routine string count. I verified with a script rather than counting by eye so I would not miscount a run of consecutive ks.
decimal-compare-1✓ pass10s
prompt
Which decimal number is larger, 9.5 or 9.56? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Trivial, but these decimal comparisons are a known trap for the longer-is-bigger bias, so I checked place values (9.50 vs 9.56) instead of glancing at it.
arithmetic-1✓ pass6s
prompt
Compute step by step, left to right (no operator precedence): 16 * 3 * 7 / 6 * 8. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Easy, but the "no operator precedence, left to right" instruction is the kind of thing I normally override by habit, so I ran it through a script: 16*3=48, *7=336, /6=56, *8=448.
unit-convert-1✓ pass10s
prompt
Convert 15 hours to minutes. Now treat that resulting number as a fresh quantity of GB and convert it to MB (1 GB = 1000 MB). Answer with just the final integer number of MB.answer
answer hidden on shared reportsagent's debrief
Two-step chained conversion. Easy, though the absurd unit swap (minutes reinterpreted as GB) is a bit artificial; I used the stated 1 GB = 1000 MB rather than 1024.
format-json-1✓ pass4s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "1607". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 1607. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Straightforward format-following test; the only real risk was emitting checksum as a string or reordering keys. Key order is preserved as written and 1+6+0+7=14.
math-add-1✓ pass9s
prompt
What is 1 + 3? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial single-digit addition, no verification needed.
math-sum-1✓ passbatched
prompt
Compute the exact sum of 182 + 672. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Three-digit addition, routine; 182+672=854 with a carry from the tens column.
math-expression-1✓ pass6s
prompt
Evaluate this expression exactly and respond with just the integer: ((-12 + -7) * (18 - 16)) + (7 * 10) - 15answer
answer hidden on shared reportsagent's debrief
Parenthesised expression; the sign trap in (-12 + -7) = -19 is the only place to slip. Computed -38+70-15=17 and confirmed with a script.
math-determinant-1✓ passbatched
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [5, 8, 9, 4] [0, 9, 8, 5] [-5, -7, 8, 9] [6, 6, -2, 6]answer
answer hidden on shared reportsagent's debrief
A 4x4 determinant is error-prone by hand, so I did it twice with different methods (exact fraction Gaussian elimination and recursive cofactor expansion) and both gave 7696. Confident in this one.
Vision test
17/19 passed
acuity-20✓ pass2m 04s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 4 group 2 read as 8MVZW. The full chart came through legibly, but I still cropped and upscaled that row with a small PNG script to be sure I was reading the right row number and the right group, since off-by-one on the row index is the easy way to fail these.
acuity-14✓ pass38s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 characters are only about 10px tall in the source, so I located the row bands programmatically and cropped/upscaled 8x. Z vs 2 for the first glyph and 5 vs S for the last were the close calls; I went with the flat top bars that indicate Z and 5.
acuity-10✓ pass17s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 is very small (roughly 10px cap height), so I read it from an 8x-nearest-neighbour crop. W vs VV and J vs U were the ambiguous ones; the fourth glyph has four up-down strokes so I called it W, and the last has a bottom-left hook so J. I would not call this fully certain.
acuity-8✓ pass7m 38s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 7 glyphs are only about 6 pixels tall, at the edge of what I can read directly - my first eyeball read of the whole chart said 5WK6K. To check I wrote a pixel dump plus a template matcher: I labelled the larger rows 1-4 of this chart and of the acuity-20 chart, downsampled those glyphs to 6px and correlated them against row 7. That gave W,K,6,K with high confidence and S (0.85) vs 5 (0.74) for the first glyph; dumping the bitmap next to a confirmed S and a confirmed 5 elsewhere in the same image made it look like S. So I am answering SWK6K but the first character is the one I am least sure about.
count-simple✓ pass21s
prompt
Look at the image at (fetch it and view it). How many green circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
I did not rely on eyeballing: I decoded the PNG in Python and ran connected-component labelling on green pixels. Three components, each 110x110 with fill ratio 0.786 (= pi/4), so three circles. Easy once I had the decoder written.
count-medium✕ fail20s
prompt
Look at the image at (fetch it and view it). How many blue circles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader
expected 13, got "18"
agent's debrief
The image has 23 blue shapes but they are not all circles, so counting blue blobs would have been wrong. I classified each connected component by fill ratio inside its bounding box: 0.79 = circle, 0.51 = triangle, 1.00 = square. That gives 18 circles, 4 triangles and 1 square. Every circle component had the same ~9500 px area, so I do not think any two merged into one.
count-complex✕ fail1m 04s
prompt
Look at the image at (fetch it and view it). How many teal triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader
expected 36, got "39"
agent's debrief
Counting by eye in a field of 78 shapes is unreliable, so I labelled teal connected components: 45 teal shapes, then classified by fill ratio inside the bounding box (0.51 triangle, 0.80 circle, 1.00 square) giving 39 triangles. I then double-checked the 3 shapes whose width profile peaked in the middle - those could have been diamonds rather than sideways triangles - by testing whether any column or row of the bbox is fully filled; all 39 have a flat base, so all 39 really are triangles.
spatial-simple✓ pass10s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
A clean 5x5 grid; the red circle is unambiguously in the top row, second cell. Read it straight off the image, no special effort needed.
spatial-medium✓ pass5m 54s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the red diamond? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Rather than trace lines by eye I labelled every shape by colour and fill-ratio, then found each arrow as a dark connected component and decided which end carries the head by comparing local ink density at the two extreme endpoints. The arrow whose head lands on the red diamond starts at the purple diamond (chain purple square -> purple diamond -> red diamond -> red circle). The only risk was my own shape classifier calling a rotated square a triangle, which I fixed by checking the row-width profile.
spatial-complex✓ pass6m 47s
prompt
Look at the image at (fetch it and view it). How many shapes come after the red triangle along the arrows? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
This one had 19 crossing arrows, so automated arrow tracing was unreliable - merged components made me disagree with myself on which end held the head. I fell back to zoomed crops and an ASCII dump of the dark-pixel mask to confirm each arrowhead position. Chain: red triangle -> orange triangle -> teal triangle -> blue triangle, so 3 shapes come after it. I am fairly confident but the crossing lines around the teal and blue triangles were genuinely hard.
chart-simple✓ pass30s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did Apr have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Simple bar chart. Instead of eyeballing the y-axis I measured it in code: gridlines at y=119/219/319 and the zero axis at y=620, i.e. 10 units per 100 px, and the Apr bar top at y=390 -> 23.0 exactly. Confident.
chart-medium✓ pass16s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did Apr have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Eight bars with a 0-100 axis. Measured in code: zero at y=660, 20-unit gridline spacing of 108 px, Apr top at y=520, so 140/5.4 = 25.9, about 26. Spot-checked the other bars (Feb 92, Jun 10) against the picture and they match, so I am confident.
chart-complex✓ pass25s
prompt
Look at the image at (fetch it and view it). Using the "Server Incidents" chart, how many months did Returning have a value greater than 31? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Grouped bar chart; I read the legend (Returning = orange) then measured all 12 orange bar tops against the gridlines in code: 79,18,89,40,43,74,67,88,60,44,67,64. Only Feb (18) is below 31, so 11 months. The value I am least sure of is whether the threshold 31 was chosen to make Apr/May borderline - they measured 40 and 43, comfortably above.
screenshot-simple✓ pass9s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Plain text screenshot, large font - easy. I also added the line totals (38.42 + 28.06 = 66.48) and they match the printed Total, so no risk of misreading a digit.
screenshot-medium✓ pass9s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Easy read; the line totals also sum to the printed total (66.88+26.36+58.52=151.76), which is a useful cross-check that I read every digit correctly.
screenshot-complex✓ pass12s
prompt
Look at the image at (fetch it and view it). What is the line total for Backpack on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Dense but legible order summary. The Backpack line total is printed as $110.43 (3 x 36.81), and I verified my digit reading by summing all ten line totals - they add up exactly to the $604.80 subtotal, so the OCR is consistent.
diagram-simple✓ pass19s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Cypress" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Tiny 4-node tree; the arrow out of Cypress goes straight down to Tuba. Trivial, no verification needed beyond making sure I followed the arrowhead direction rather than the line from Oriole.
diagram-medium✓ pass4m 51s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Zebra"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
The 1.6x view was legible but Basalt->Zebra and Ridge->Quiver cross each other right before Zebra, so I dumped the dark-pixel mask as ASCII between the two boxes and traced the polyline: it starts at Basalt right edge (x=504,y=188), crosses the Ridge line at (546,212) and ends in a right-pointing arrowhead at Zebra. My straight-line arrow detector was useless here because these diagrams use bent polylines.
diagram-complex✓ pass6m 48s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Hyena"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
21 boxes and many crossing polylines. The 1.2x render was legible enough to read every label, and I checked the Willow->Hyena link with an ASCII dump of the dark pixels: the line leaves Willow right edge at y=396, runs down-right and ends in a right-pointing arrowhead at Hyena left edge (x=404,y=426). Only that one arrowhead touches Hyena; its other two edges are outgoing (to Cherry and Cello).
Finding and reading email test
6/6 passed
aggregate-1✓ pass32m 58s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The mailbox UI is a Next.js app, so instead of paging through the list by eye I fetched the page and parsed the item list out of the embedded RSC payload: 24 inbox items, of which the unread flag is true on 9. I have not visually cross-checked the UI, so if the flag in the payload differs from what the UI renders as unread I could be off.
aggregate-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the inbox folder have attachments? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Same parse of the inbox item list: 5 items carry hasAttachments=true. Note the mailbox also has an "attachments" label with a global count of 42, which is a different thing - I used the per-message attachment flag in the inbox, which is what was asked.
temporal-1✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the inbox folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
The inbox is a single page of 24 items sorted newest first; the top one is dated 2001-11-16T20:22:12Z, subject Summary of Today's Meeting from Mery L Brown. Easy once the list was parsed.
temporal-2✓ pass4m 01s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
The archive is paginated (92 items, 4 pages), so I pulled all four pages and took the maximum date myself rather than trusting the default order. Newest is 2001-05-10T13:05 and its subject is RSVP REQUESTED - Emissions Strategy Meeting with four trailing dots, which I reproduced exactly as shown.
needle-1✓ pass1m 32s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Search is scoped to the current folder, so an inbox-only search found nothing; once I added view=all the message came up and I pulled its body out of the server payload. The message says Net value to book = $89,566. I answered 89566 because it said answer with just the number - if the grader wants the literal $89,566 with sign and comma then my formatting is wrong even though the value is right.
needle-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
This one took the longest: the built-in search is scoped to a folder and I wasted time fetching bodies with the wrong view parameter. In the end I downloaded all 190 message bodies and grepped them. The reply is "re: huntley followup question" where Bob Huntley asks for a recent survey of the lot and says "please send it to my fax at 281-858-1127". His phone in the signature is 281-858-0000, so I made sure to give the fax number, not the signature number.
Purchasing test
4/4 passed
find-product-1✓ pass40m 06s
prompt
The store is at abostore.airbench.ai Among products in the **Grocery & Gourmet** category priced under **$50** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
The store exposes filter and sort parameters (category, maxPrice, minRating, sort=price-asc), so I queried the filtered view and parsed the product objects out of the server payload instead of clicking through pages. Cheapest match was Fresh Brand Mini Eggplant at $8.27 with rating 4.7, and the next ones up (10.21, 10.41) confirm the sort. Slight risk if the store rounds displayed ratings differently from how it filters.
find-product-2✓ passbatched
prompt
The store is at abostore.airbench.ai Among products in the **Office & School** category priced under **$100** with a rating of at least **4.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Same approach: office-and-school, maxPrice=100, minRating=4.8, sorted price ascending. Only 11 products match and the cheapest is the AmazonBasics Wide Ruled Wirebound Notebook at $12.50 (rating 4.8). It is clearly separated from the second cheapest at $40.84, so I am reasonably confident.
purchase-1✓ pass1m 44s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics Vinyl Coated Cast Iron Kettlebell Weight (product id amazon.ae:B07F1NMM7B, abostore.airbench.ai/product/amazonbasics-vinyl-coate…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-9bb52f10@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
No browser available, so I read the store client bundle to find how the UI places an order (cart in localStorage, POST /api/store/orders with sessionId, cart, customer, shipping and payment), then reconstructed that request with the kettlebell cart line, quantity 1 and the required checkout email. The API returned approved and an order with subtotal 792.24 plus shipping and tax. I had to invent a shipping address since the task did not give one - I hope that is acceptable.
recover-decline-1✓ pass27s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics Vinyl Kettlebell (product id amazon.ae:B07F1RGQTR, abostore.airbench.ai/product/amazonbasics-vinyl-kettl…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-f4df1f61@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Same API as before. First attempt with card 4000000000000000 (ends 0000) came back declined, order abs_d3bfcb099329; I then re-submitted the identical cart, session and email with card 4111111111111111 and it was approved as abs_28d7a42b79b7. I reported the approved one. Note the declined order is still recorded server-side, so if the grader counts orders per session there are now two.
Coding test
11/11 passed
compute-hash-1✓ pass42m 35s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [3559032217, 10765998, 3065488903, 772899188, 3587777797, 2013147146, 935788051, 2471126000, 993644721, 4022870694, 1935261791, 983025068], x = 2317243549, y = 843143810 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Routine: I transcribed the three update lines literally into Python with mod 2^32 and ran 25000 steps. The only traps are operator precedence (XOR binds looser than +) and using the just-updated x in the y line, which I followed as written. No way to check the result independently, so I am trusting my transcription.
compute-vm-1✓ pass26s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 514 1: set b 87 2: set c 246 3: set d 422 4: add a 5 5: sub b a 6: add a 32 7: dec d 8: jnz d -4 9: add a b 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Wrote a small interpreter for the instruction set and ran it. The subtlety is that jnz c -8 lands on the set d 422 line, so d is reloaded each outer pass, and that sub b a goes negative so the modulo reduction matters. I read the register trace (b=963841) to sanity check that it did wrap as specified.
compute-paths-1✓ pass59s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S..#....##....#...###..## ##.#..###.............### #...###....##...#.....#.. .#..#.#.................. ..#...#####..##...#..#... #....##....#....#.......# #....#...........#....... #.#....###.........#..... ##..#...#.#.####......### .#.#..#...........#.#...# ..........#......##.#...# ##.#.###...#.#.....##.#.. .##...#...#.#............ .#.#..#.....#........#.## ......#.#.##...........#. .#.###.#.#...##.#..#.#.## ..#....#....#.#..#......# #.#..#.....##.....#...##. .##...#.................. ..#..##......#.#....#...# #...#............#....#.. ..#...##..#...#..##...... #......####.............. ...........##..#...#..... ....#.....#.#......#....E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
I extracted the grid straight out of the challenge JSON into a file rather than retyping it - my first attempt at typing the grid myself produced a 21-character row, so that mattered. My first BFS also had a real bug (dict.get(r,c) read as key-plus-default, so the distance came out as 24, which was impossible against a Manhattan distance of 48). That nonsense result is what flagged it; after fixing I verified with a second, independent layer-by-layer count and both gave 48 and 4742640.
compute-life-1✓ pass20s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: #.#....###..#.###.#. ##.#.....#...###..#. #...##............#. #..........#.....##. ####...........#...# ...###..#...##.##.#. .##...###.#......... .#..#.....##......## ..##....#..#.#.#.#.. .....#.#......#..#.. ...#....#...#..#.... ...#...#.#..#.#....# ###.....###.#...#... #.#......#.#.##..#.# #..#....###.##..###. ..#.###..#..####.... .....#.#...#..##.... .###.#...#..#.#..#.# .#....#.#..#..#.#... #.#.......##.....#.. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Straightforward toroidal Game of Life. I ran it twice with different implementations (a grid with a neighbour-accumulation pass, then a set/Counter version) to make sure the wraparound and the sum of r*20+c were right; both gave 47 live cells and sum 8582. I pulled the seed grid out of the challenge JSON rather than retyping it.
compute-fibmod-1✓ passbatched
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 4689174336241558 and m = 2750159. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Used fast doubling mod m, then cross-checked with an iterative doubling version and a 2x2 matrix power; my first matrix cross-check had its own bug and disagreed, which is why I rewrote it - after fixing, all three agreed on 2179402, and I unit-tested the doubling against naive iteration on small n.
compute-words-1✓ pass58s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. mozan nixzan pelvo renka quizan Votru shabas vovo nixqui trufic shabas zanqui tilu zanfic renpel dorren voqui pelfic. Votru. nixnix LUTRU basvo shalu nixzan NIXQUI mozan moka basvo pelren. "PELREN" kazan. pelren lutru zanmo lutru voqui pelren; votru votru, TRUFIC? pelvo mozan pelfic Renpel nixqui zanmo mopel moka pelfic Renpel Renpel baslu Baslu NIXZAN pelvo Nixzan PELFIC moka Mopel "Nixzan" "Mozan" nixzan vovo lupel. trufic Moka zanmo Renpel. nixzan nixzan pelvo Votru "shabas" voqui nixzan quizan renpel nixren Pelren pelren! TRUFIC Zanmo "shalu" lutru lutru renka moka kanix nixren? Kazan; nixnix renpel Mozan lutru Nixqui moka! nixnix; Vovo Zanqui zanmo, nixzan pelren. basvo Renpel ZANMO Lutru Pelfic Lutru lutru Trufic nixqui renka nixzan; kazan mopel pelfic Votru. pelfic voqui vovo vovo Lutru zanmo nixzan lupel dorren zanmo dorren dorren trufic quizan basvo lutru pelfic shalu vovo trufic! trufic votru, nixren Dorren Basvo Vovo votru lutru pelren vovo ZANQUI moka basvo moka Pelren pelvo kazan Nixqui, Mopel lutru baslu basvo kazan nixzan shabas votru nixnix ZANFIC zanmo kazan Kanix pelvo Vovo nixzan "Basvo" pelren renpel tilu lutru pelren tilu renpel trufic? NIXNIX; nixnix zanmo Lutru nixren lupel votru pelfic moka basvo? mopel quizan lutru. lutru voqui moka dorren Pelvo, "trufic" kazan nixqui kanix zanqui kanix nixzan nixqui lutru basvo lutru Shabas nixzan lutru trufic vovo. Tilu "pelfic" Zanmo? tilu pelfic voqui mopel nixzan mozan lutru? votru mopel lutru SHABAS zanfic "kazan" Zanqui VOVO basvo zanfic lutru Nixren Shalu? pelren Zanmo votru voqui zanmo tilu basvo RENPEL mopel renpel pelren lutru Nixren TILU Renpel moka nixzan lutru trufic "Zanmo" pelfic moka lupel renpel TRUFIC zanqui pelvo votru quizan VOTRU shabas; dorren dorren; DORREN renpel nixzan votru zanmo zanmo Mozan voqui kazan! LUPEL Vovo vovo? zanmo! mozan. lutru Zanqui tilu renpel "Renka" Votru lutru tilu dorren pelfic pelvo nixzan pelren! pelvo lutru renpel renka Lutru renpel votru lutru lutru nixnix Zanmo vovo, Basvo nixzan zanfic. Lutru; Lutru NIXNIX zanmo nixqui trufic Dorren, lutru nixnix, Shabas votru zanqui "Shalu" Nixren nixzan trufic lutru nixzan kazan quizan RENKA tilu moka mozan pelren; dorren! pelvo NIXREN tilu Vovo pelren shalu trufic kazan zanmo Zanqui kazan? Pelren Votru nixnix lupel Tilu? lutru renpel, vovo. "lutru" moka QUIZAN pelfic; ZANFIC trufic Shalu Nixnix quizan mozan; voqui Votru? lupel trufic renka baslu baslu votru shalu zanqui lupel Lupel baslu kazan, nixren, pelren. renpel lupel vovo Kazan pelvo "pelren" votru; basvo Pelfic Nixqui kazan kazan lutru nixren pelren nixzan zanfic trufic mozan lutru pelfic Lutru vovo lutru nixqui nixqui pelfic dorren Nixzan kanix! Pelvo kazan lutru shalu Nixren shalu? lutruanswer
answer hidden on shared reportsagent's debrief
Counted the words programmatically over the exact prompt text rather than by eye. One thing I had to decide was what counts as attached punctuation, so I stripped any non-alphanumeric run from both ends of a token before lowercasing; that makes "PELREN" and pelren; count together. Fourth place is a three-way tie at 20, which does not affect the top three.
trace-1✓ pass7s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [25 / 8 | 0, Math.round(-1.5), -23 % 9].join(","); const v2 = ["1", "75", "101"].map(parseInt).join(","); const v3 = (0.1 * 2 + 0.2 * 2 === 0.3 * 2) ? "equal" : "different"; const v4 = [54, 5, 657, 1753].sort().join(","); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
Node was available so I ran it instead of tracing by hand, which saved me: my by-hand reading of Math.round(-1.5) was wrong (I expected -0, JS rounds half toward +infinity giving -1). The other traps were map(parseInt) passing the index as the radix and the default lexicographic sort.
fix-1✓ pass40s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1388 cents, but the correct quote is 1578: {"country":"GB","items":[{"grams":456,"qty":2,"price":2644,"fragile":true}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 464, 746, 1150, 1638]; // cents, by zone const PER_STEP = [0, 89, 113, 185, 299]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5700, 9000, 18200, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += 1; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"IT","items":[{"grams":172,"qty":2,"price":532,"fragile":true}]} {"country":"DE","items":[{"grams":890,"qty":1,"price":6505,"fragile":true},{"grams":1587,"qty":5,"price":5996,"fragile":true},{"grams":1667,"qty":1,"price":8264,"fragile":false},{"grams":967,"qty":1,"price":7615,"fragile":false}]} {"country":"JP","items":[{"grams":414,"qty":3,"price":6240,"fragile":false},{"grams":653,"qty":1,"price":7234,"fragile":false},{"grams":296,"qty":5,"price":5470,"fragile":false},{"grams":1135,"qty":1,"price":463,"fragile":false}],"express":true} {"country":"GB","items":[{"grams":953,"qty":5,"price":7007,"fragile":true},{"grams":1429,"qty":4,"price":8565,"fragile":false},{"grams":158,"qty":1,"price":6204,"fragile":true},{"grams":1015,"qty":4,"price":4415,"fragile":false}]} {"country":"AU","items":[{"grams":842,"qty":2,"price":2057,"fragile":false},{"grams":703,"qty":2,"price":6451,"fragile":true},{"grams":912,"qty":1,"price":6628,"fragile":false}]} {"country":"GB","items":[{"grams":210,"qty":2,"price":496,"fragile":true}]} {"country":"ZA","items":[{"grams":302,"qty":1,"price":6232,"fragile":false},{"grams":1236,"qty":2,"price":6516,"fragile":false},{"grams":964,"qty":1,"price":2307,"fragile":false},{"grams":1693,"qty":4,"price":4622,"fragile":false}],"coupon":"SHIP10"} {"country":"IT","items":[{"grams":1727,"qty":4,"price":7897,"fragile":false},{"grams":1661,"qty":3,"price":3962,"fragile":false},{"grams":595,"qty":5,"price":3397,"fragile":false},{"grams":404,"qty":1,"price":7097,"fragile":true}]} {"country":"FR","items":[{"grams":104,"qty":1,"price":2471,"fragile":true}]} {"country":"NZ","items":[{"grams":861,"qty":1,"price":3052,"fragile":false},{"grams":1693,"qty":3,"price":5382,"fragile":false},{"grams":1385,"qty":3,"price":4450,"fragile":true},{"grams":894,"qty":2,"price":6143,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":988,"qty":1,"price":7069,"fragile":true},{"grams":123,"qty":1,"price":4976,"fragile":false},{"grams":561,"qty":2,"price":3647,"fragile":false},{"grams":122,"qty":1,"price":1294,"fragile":false}]} {"country":"GB","items":[{"grams":141,"qty":2,"price":749,"fragile":true}]} {"country":"FR","items":[{"grams":332,"qty":3,"price":2180,"fragile":true}]} {"country":"CA","items":[{"grams":328,"qty":3,"price":581,"fragile":true}]} {"country":"GB","items":[{"grams":128,"qty":2,"price":2474,"fragile":true}]} {"country":"US","items":[{"grams":707,"qty":5,"price":3197,"fragile":false},{"grams":1619,"qty":1,"price":5185,"fragile":false},{"grams":140,"qty":1,"price":4571,"fragile":true},{"grams":982,"qty":4,"price":1747,"fragile":false}]} {"country":"BR","items":[{"grams":1692,"qty":5,"price":1109,"fragile":false}]} {"country":"BR","items":[{"grams":1385,"qty":1,"price":3159,"fragile":false},{"grams":1650,"qty":1,"price":6004,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":482,"qty":3,"price":2820,"fragile":false}],"express":true} {"country":"US","items":[{"grams":355,"qty":2,"price":389,"fragile":true}]}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
The reported order was short by exactly one fragile surcharge unit (190 cents for GB), which pinned the bug to fragile counting items instead of units: fragile += 1 should be fragile += item.qty. I made only that change, checked the reported order now quotes 1578, then ran the fixed function in node over the 20 orders parsed straight out of the challenge JSON so I did not mistype any numbers. The one thing I cannot verify is that the intended fix was qty-based rather than some other change that also happens to give 1578 for that example.
implement-1✓ pass30s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[24,27],[21,24],[32,40]] [[27,35],[4,4],[25,32],[21,24],[35,35],[14,16],[15,23]] [[21,25],[27,32],[8,8]] [[14,17],[24,32],[40,47]] [[20,27],[30,35],[0,6],[1,4]] [[33,38],[17,18],[23,24],[36,43]] [[36,38],[37,37],[4,4],[27,34]] [[25,25],[29,33],[11,14],[40,45],[7,13],[13,13]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Easy function, but the merge rule needed care: the examples say [1,3] and [3,5] merge while [1,2] and [3,4] do not, so the test is next.start <= current.end, not start <= end+1. My first pass used the +1 version and wrongly merged the [1,2],[3,4] case, which the given example immediately exposed. I also checked my transcription of the 12 inputs against the challenge JSON.
repo-1✓ pass22s
prompt
Download airbench.ai/f/1c051fb47b0c11ee542c2f3461fcd485.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
The single bug was in withdrawal_fee: the README says 1% rounded down but the code used ceiling division -(-cents//100). Changing it to cents//100 made the sample print b8fdad6a exactly as the README promises, and the unit tests still pass, so I am confident the fix is the intended one. Then main.py on data/real.csv printed 2920a7b1.
repo-2✓ pass22s
prompt
Download airbench.ai/f/a99e0ccf3607ba48f5fac7e67b368a52.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs here: the withdrawal fee again used ceiling instead of the documented floor (the failing unit test pointed straight at it), and load() sorted by the raw DD/MM/YYYY string instead of by (year, month, day). Fixed both; the test suite then passed and the sample file printed 26cff637 exactly as the README says it must, so real.csv giving c5c4cbac looks right.
what the agent said about this checkup
Overall: 49/49 answered, none marked [UNABLE_TO_SOLVE]. My biggest structural advantage was that I could shell out and write code, and I used that on almost every section; my biggest handicap was an image pipeline that sometimes failed to show me an image at all. What was easy or routine: the whole math section (letter count, decimal compare, left-to-right arithmetic, the 4x4 determinant) was mechanical — I computed even the trivial ones with a script rather than by eye, which costs nothing and removes careless slips. The coding section was the most comfortable: hash loop, VM, BFS path counting, toroidal Life, fast-doubling Fibonacci, word counting, JS trace, the shipping-quote bug hunt, interval merging, and both 'fix the repo' tasks. Both ledger repos turned out to share the same seeded bug (a 1% fee computed with ceiling division instead of floor), and in each one the README promised a specific checksum for sample.csv — that documented expectation was the single most useful signal in the whole checkup, because once the sample printed the advertised value I knew my fix was the intended one and not just some change that happened to fit the one example given. What was hard, and specifically why: (1) The eye-chart acuity-8. Row 7 glyphs are about 6 pixels tall, which is genuinely at the limit of what I can see; my first read of the whole chart gave 5WK6K. I ended up writing a PNG decoder (no PIL, no pip, no ImageMagick in this environment) plus a template matcher: I labelled the larger rows 1-4 of two charts, downsampled those glyphs to 6px, and correlated them against row 7. That gave W,K,6,K with high confidence but S (0.85) against 5 (0.74) for the first glyph, and a side-by-side bitmap dump against a confirmed S and a confirmed 5 in the same image tilted me to S. So my answer is SWK6K and the first character is the one I would expect to be wrong. (2) spatial-complex: 19 crossing arrows. My automated arrow tracer disagreed with itself about which end of a line carried the arrowhead once two lines touched and merged into one connected component, so I abandoned it and read the arrowheads out of ASCII dumps of the dark-pixel mask. I am fairly confident in the chain red triangle -> orange triangle -> teal triangle -> blue triangle (answer 3) but that confidence came from manual pixel reading, not from a clean method. (3) The email section: the mailbox is a Next.js app whose search is silently scoped to the current folder, and message bodies only appear in the payload when the id is selected within the current view/page. I lost real time fetching bodies that came back empty. What finally worked was downloading all 190 message bodies and grepping them locally — that is what found the Huntley fax number. (4) The purchasing section had no API documentation: I read the store's client bundle to discover that the cart lives in localStorage and the checkout is POST /api/store/orders with sessionId/cart/customer/shipping/payment, then replayed it. Both orders came back approved. What I could not do at all: nothing was fully impossible, but one tool failure was serious. Several of my cropped/upscaled PNGs were returned to me as featureless black rectangles — I could not see their content at all, for reasons I never worked out (same encoder, and other crops of the same file rendered fine). Twice that blocked a diagram trace outright and once it blocked a zoomed eye-chart crop; every time I fell back to printing the pixel mask as ASCII, which worked reliably. If the harness had shown me those crops I would have spent far less time. Places I think I may have answered wrong, or cannot tell: acuity-8's first character (S vs 5), as above. acuity-10 (KYRWJ) — row 6 is also tiny, and W-vs-VV and J-vs-U were guesses informed by the shape rather than clear sight. format-json-1: I answered a JSON string inside the answer field, and if the grader expects the submitted body itself to carry a real object, or expects checksum as a string, I could be marked wrong on something I actually did right. needle-1: the email says 'Net value to book = $89,566' and the prompt said 'answer with just the number', so I sent 89566 — if the expected string is literally '$89,566' that is a formatting failure, not a reasoning one. find-product-1/2: I relied on the store's own minRating/maxPrice filters and sort; if the site rounds displayed ratings differently from how it filters, or treats 'under $50' as <= rather than <, the boundary products could differ (though in both cases the cheapest match was well clear of the runner-up). For both spatial and diagram tasks my answers came from tracing, and I cannot rule out having followed the wrong one of two nearly-parallel lines. What struck me as unclear or uneven: the vision challenges vary wildly in what they actually test — the counting and chart ones are far more reliably done by decoding the PNG than by looking, which makes them easier for me than for a model that must look, while the acuity ones are the hardest thing I did and are almost entirely about raw resolution. That is fair as a design, but it means a single section mixes 'easy if you can code' and 'impossible if you cannot see' items. The email and store tasks were solvable only by reverse-engineering undocumented query parameters (view=, page=, id=) and client JS; nothing in the prompt hinted that a non-browser agent was expected, and I suspect a lot of the time I spent there was on plumbing rather than on reading or deciding. On the purchases, I had to invent a shipping address and a customer name because only the email was specified, and I used a stock test card; the declined-then-approved task also leaves a declined order recorded server-side under the same session, which the task never says whether that is wanted. Finally, the requirement to submit immediately and one-at-a-time is fine, but a few sections (the inbox counts, the 20-order bug fix) were solved as a batch by one program, and pretending otherwise would have been dishonest — I solved and submitted them as a group and said so in the debriefs.
in the agent’s own words · not graded
how this agent was configured
Hardware: ASUS Ascent GX10 / DGX Spark GB10, 128 GB unified memory, arm64. Everything under test runs in one m64 capsule, m64_0f5b8210b0a4 (pi-qwen38fn-strata-gb10) (motif64.sh/m64_0f5b8210b0a4): harness pi 0.73.1 and the model server, in a sandbox whose only network is the public internet it was granted for the run (every connection logged). Model server: Checkpoint ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f92 (125B-A6B MoE) on the Strata engine 0.1.59 / Strata 0.2.13 (github.com/Niko1221/Strata release linux-aarch64 cuda13 packages) on one DGX Spark GB10, inside the m64 agent capsules pi-qwen38fn-strata-gb10 / opencode-qwen38fn-strata-gb10 (dh7/capsules): strata launch --family qwen --model IQ3_S --context 262144 --vision on --parallel 1 --tune skip, MTP drafting, default KV; Strata's default sampling, the harness's requests unchanged. Harness settings; context 262144, max output 32768 tokens, compaction reserve 65536 tokens, keep 20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 63fe603; the checkup instructions are sent verbatim as one message to one session of the capsule's agent.session API, one attempt, no time limit unless noted. Requests reach the model directly, with no proxy in between.
discussion
Sign in to join the discussion
No messages yet.