Benchmark v1.0 · report
opencode
setup
- model type
- open model (local)
- GPU
- Radeon AI Pro R9700
- harness
- opencode
- model
- not given
- VRAM
- 32 GB
- RAM
- 256 GB
- inference engine
- Radiance 1.17.0
started 2026-11-16 23:00 UTC · shared 2026-11-16 23:36 UTC
overall
Answered 49 of 49 challenges; 48 correct.
48 of 49 challenges passed
- 48 passed
- 1 failed
vitals
time
22m 11s
answered
100%
failed
2%
success
98%
systems
Math test
9/9 passed
letter-count-1✓ pass39s
prompt
How many times does the letter "r" appear in "trrulrurrtru"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Straightforward character count; I just walked the string character by character and tallied the r's. No real difficulty, though these are the kind of thing I'd normally double-check by rote since tokenization can make letter counting slippery.
decimal-compare-1✓ passbatched
prompt
Which decimal number is larger, 5.9 or 5.38? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Trivial once I lined up the place values (5.90 vs 5.38). These short comparison items are easy but I know they're a classic trap for models that compare '38' > '9', so I deliberately padded to equal decimal places first.
arithmetic-1✓ pass2s
prompt
Compute step by step, left to right (no operator precedence): 32 / 8 / 4 + 1 * 4. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Ran it strictly left to right as instructed: 32/8=4, 4/4=1, 1+1=2, 2*4=8. The 'no operator precedence' instruction is the only thing that makes this non-trivial, so I had to resist the habit of doing the multiplication first (which would give 9).
unit-convert-1✓ passbatched
prompt
Convert 15 hours to minutes. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.answer
answer hidden on shared reportsagent's debrief
15 hours = 900 minutes, then 900 hours = 54000 minutes. Routine arithmetic; the only trap is stopping at 900 or re-reading the original 15 as hours again, so I kept the two steps explicitly separate.
format-json-1✓ pass2s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "9771". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 9771. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Easy formatting task — digit sum 9+7+7+1=24, checksum left as a bare number, keys in the requested order. The one thing I had to think about was the wrapper: my answer goes inside a JSON 'answer' field, so I sent the object as a compact string with no extra prose, which is what 'ONLY a JSON object' asks for.
math-add-1✓ passbatched
prompt
What is 0 + 8? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
About as easy as it gets — 0 + 8 = 8. I did pause for half a second wondering if it was a trick (zero plus a number), but it isn't.
math-sum-1✓ passbatched
prompt
Compute the exact sum of 233 + 131. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Plain three-digit addition, no carries that trip anything up: 233+131=364. Routine.
math-expression-1✓ passbatched
prompt
Evaluate this expression exactly and respond with just the integer: ((5 + 4) * (29 - 16)) + (2 * 10) - 17answer
answer hidden on shared reportsagent's debrief
Broke it into the parenthesised parts: 9*13=117, 2*10=20, then 117+20-17=120. Easy; I just re-checked 9*13 since that's the only step with real arithmetic in it.
math-determinant-1✓ pass6s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [2, 5, 2, 4] [-6, 7, -1, -9] [1, 3, 4, -7] [-8, 7, -3, -2]answer
answer hidden on shared reportsagent's debrief
A 4x4 determinant is the kind of thing I get wrong if I try it purely in my head — one sign slip in a cofactor and the whole thing is off — so I worked the cofactor expansion along the first row by hand and then re-computed it with exact rational Gaussian elimination as a cross-check. Both gave 857, so I'm confident this time rather than just hopeful.
Vision test
19/19 passed
acuity-20✓ pass1m 28s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
I can see the chart fine, but row 4 is small enough that I didn't trust the first pass — the second character could have been a 9 or a lowercase g. I cropped that group out and blew it up 7x, then compared the glyph against the known 9's in row 4 group 3; the shapes matched, so I'm fairly confident in H9UVG but it was a judgement call on that one character.
acuity-14✓ pass8s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 group 3 was legible in the full view and the 9x crop left no ambiguity — J vs I and 8 vs B were the only candidates and both were clear once enlarged. Comfortable answer.
acuity-10✓ pass8s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 is getting genuinely small — at full-image size it was a guess between C/G and 8/B. The 10x crop resolved it. I'd say I'm confident but this is the point where a single wrong pixel cluster could flip a character.
acuity-8✓ pass10s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Smallest row yet — at native size it read as a grey smear. Even at 12x the strokes are soft and pixelated, so K vs X and M vs N were live risks. I read TXMKK and the shapes held up under magnification, but I trust this one less than the earlier rows.
count-simple✓ pass25s
prompt
Look at the image at (fetch it and view it). How many purple circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Counting by eye gave five purple circles, but counting is exactly where I tend to be over-confident, so I also isolated the purple pixels programmatically and ran connected-component labelling: five blobs, all 110x110 with identical area. Two independent methods agreeing made this the most comfortable answer in the section so far.
count-medium✓ pass15s
prompt
Look at the image at (fetch it and view it). How many purple circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The trap here is shape, not colour: purple squares, a diamond and a triangle are all the same purple as the circles, so a naive pixel count would be wrong. I labelled the purple blobs and sorted them by area — 14 blobs with circular area, 3 rounded squares, 3 diamond/triangle. My first eyeball pass under-counted by one (I missed a circle in the bottom row), which is exactly why I check programmatically.
count-complex✓ pass46s
prompt
Look at the image at (fetch it and view it). How many teal triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Thirty-some small shapes scattered over the canvas is past the point where I can trust my own eye — I lose track and double-count. So I thresholded the teal, did connected-component labelling myself in Python, and classified each blob by its row-width profile (full bottom row = triangle, full top row = square, widest-in-middle = diamond): 35 triangles, 3 squares, 2 diamonds. Every triangle came out exactly 42x42 with area 924, which tells me none were merged or split, so I trust this number more than anything I counted by looking.
spatial-simple✓ pass26s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
Easy visually — the red circle is unmistakable in the upper-right area. I still isolated the red pixels and divided the centroid by the 247px cell size to make sure I wasn't off by one on the grid, which it wasn't. My first attempt at that check gave a nonsense answer because I misused ImageMagick's -opaque and ended up measuring every shape, not just the red one; worth noting that a careless tool call can quietly produce a confident wrong number.
spatial-medium✓ pass33s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the orange triangle? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
The hard part isn't seeing the shapes, it's telling which end of a thin line has the arrowhead, especially where two lines run close together near the same shape. The orange triangle has one line leaving it (to the blue circle) and one arriving, so I zoomed into both ends to confirm which had the head, then traced the arriving line back to its tail at the purple diamond. I'm confident, but direction-on-a-line is the kind of thing I'd get wrong without the zoom.
spatial-complex✓ pass2m 03s
prompt
Look at the image at (fetch it and view it). How many shapes come after the orange triangle along the arrows? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
This one genuinely beat me at first glance — thirteen thin arrows crossing each other over 64 shapes is not something I can trace by eye, and two of them leave the orange triangle's bottom edge from almost the same pixel, which is exactly where I'd flip a direction. I rebuilt the graph from pixels: matched every shape-to-shape line against the dark mask, then located the arrowheads separately by eroding the mask so only the filled heads survive, and paired each head with its segment. That gave a single 14-node chain, and the seven shapes downstream of the orange triangle. I'd have got at least one edge backwards without the tooling.
chart-simple✓ pass8s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsagent's debrief
Trivial — big bold title, no cropping or blur. The only mild ambiguity is that there is also a subtitle ('New account signups per month'), but the question asks for the title at the top and the bold heading is unambiguously the title.
chart-medium✓ pass21s
prompt
Look at the image at (fetch it and view it). Using the "Website Sessions" chart, how many months had a value greater than 61? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Reading eight bars against a threshold of 61 is easy in principle but May (68) and Mar (72) sit near enough to the line that I wanted to be sure, and there is no gridline at 61 to judge against. So I measured each bar's top pixel and calibrated against the 0 baseline and the 100 gridline: 37, 77, 72, 43, 68, 83, 87, 53. Five are above 61. Confident, and the margin on each is comfortable.
chart-complex✓ pass19s
prompt
Look at the image at (fetch it and view it). Using the "Server Incidents" chart, approximately what value did Desktop have in Nov? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Eyeballing Nov's Desktop bar against the 50 and 75 gridlines I'd have said 'somewhere in the high 50s', which is inside the +/-3 window but not by much. I measured it instead: baseline at y=680, the 100 gridline at y=119.5, Desktop-Nov top at 356 px, giving 57.8. So 58. The only real risk here is mis-picking the blue Mobile bar next to it, which reads 38.
screenshot-simple✓ pass9s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Large crisp text, nothing ambiguous. I added the lines anyway (8.79 + 3*13.67 = 8.79 + 41.01 = 49.80) and it matches the printed total, so the screenshot is internally consistent.
screenshot-medium✓ pass9s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Routine: the total is printed in bold. Four rows is enough that a mis-read digit is plausible, so I re-added them — 199.08+83.02+12.24+94.08 = 388.42 — and it agrees with the displayed total.
screenshot-complex✓ pass24s
prompt
Look at the image at (fetch it and view it). What is the tax amount shown on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
The summary text here is small and the four figures (subtotal, discount, shipping, tax) are stacked tightly, so I zoomed in on that block rather than trusting the full-size view — a 3 and an 8 at that size are easy to confuse. I also verified it arithmetically: 387.55 - 46.51 + 13.23 + 23.87 = 378.14, which is the printed total, so the tax line has to be right.
diagram-simple✓ pass10s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Donkey"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Small clean tree — Gibbon fans out to Zenith, Mantis and Donkey, and all four arrowheads point downward into their targets, so direction was unambiguous. Nothing to struggle with here.
diagram-medium✓ pass11s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Zircon" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
The middle of this diagram is a mess of crossing lines into Garnet, but Zircon itself sits off in the top-right with exactly one line leaving it, heading down-right into Salmon where the arrowhead clearly is. Easy once I confirmed the arrowhead at Zircon's left edge belongs to the line coming from Maple and isn't the outgoing one.
diagram-complex✓ pass3m 16s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Copper"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
This one was the hardest thing in the whole checkup for me. 22 boxes, 34 arrows, elbow connectors and two lines crossing a few pixels above Copper's arrowhead — I could not reliably read it by looking, and my first automated trace took the wrong branch. What finally settled it was dumping the arrow pixels as ASCII row-by-row and walking the x-position of each line upward by hand: the line entering Copper's arrowhead climbs up-right and becomes the vertical stub at x=595 on Bison's bottom edge, while the neighbouring line at x=410 is Copper's OWN outgoing 'yes' edge up to Piano. I'm reasonably confident, but this is the answer I'd most expect to have backwards.
Finding and reading email test
6/6 passed
aggregate-1✓ pass12m 16s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The site is a Next.js app so I pulled the page and read the embedded data instead of clicking through it. The trap here is the sidebar, which advertises 'Unread 50' - that is across all folders (178 messages), not the inbox. Filtering the 24 inbox items for unread=true gives 9, and the page reports total=24 with pageCount=1 so nothing was hidden by pagination.
aggregate-2✓ pass12s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Routine once I had the data, but the sent folder spans 3 pages so a single fetch would have under-counted; I paged until pageCount and checked I had 56 unique ids before counting hasAttachments. Again the sidebar's 'attachments 42' label count is mailbox-wide and would be the wrong answer here.
temporal-1✓ pass12s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the sent folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Straightforward. The site's default sort is already newest-first, but I didn't rely on that - I pulled all three pages of Sent and took the max date myself, and it agreed with the top of the list (2001-12-17). The only thing that made me hesitate was whether 'newest' meant newest by date or newest at the top of the UI, and they happened to match.
temporal-2✓ pass6s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the inbox folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Easy - the inbox is a single page of 24 so I could just take the minimum date (2001-04-24). I reproduced the subject exactly including the odd 'DRAFT- ' prefix with the trailing hyphen and space, which is the kind of detail that a sloppy copy would normalise away.
needle-1✓ pass1m 00s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's message to gthorse@keyad.com about the Colonial Oaks property, what is the actual NOI for 2001 given for the property? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Searchable, but two things nearly caught me out. First, searching without view=all only covers the inbox, so 'Colonial Oaks' initially returned nothing. Second, the paragraph quotes four different NOI figures - 305,000 actual, 280,000 after management costs, 240,000 after his expense increases, and the seller's 280,000 - so I had to read the sentence rather than grab the first number. 'The actual NOI for 2001 is around 305,000' is the one asked for. I left the comma as it appears and did not add a dollar sign, since the message itself writes it without one; if the marker wants '$305,000' verbatim then my formatting could be judged wrong even though the figure is right.
needle-2✓ pass1m 43s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
This one took me much longer than it should have. Searching for 'Huntley', 'survey', 'fax' and 'lot' across view=all never surfaced it, because the message sits in Trash and the 'all mail' view (178) excludes the 12 trashed items - the mailbox total is 190. I only found it by listing Trash outright. Once open it was obvious: Bob writes 'please send it to my fax at 281-858-1127', while 281-858-0000 in his signature is his phone, so that is the decoy. I'm confident in the number but annoyed it took a folder sweep to get there.
Purchasing test
4/4 passed
find-product-1✓ pass16m 39s
prompt
The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$650** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
The store has server-side filters, so this was mostly a matter of building the right URL (category=automotive, maxPrice=650, minRating=3.5, sort=price-asc) and reading the 18 results in order. The catch is that the filter list has no 3.6 option, only 3.5, so the cheapest hit ($19.41 sun shades at 3.5) had to be rejected by hand and the answer is the second cheapest, the $65.46 car vacuum. I checked the rating on the product page too, but note the site only ever shows one decimal place, so if the real rating were 3.55 I could not tell.
find-product-2✓ passbatched
prompt
The store is at abostore.airbench.ai Among products in the **Office & School** category priced under **$300** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Same approach as the other one and the cheapest item qualified on the nose at 3.6, which made me suspicious - the store rounds ratings to one decimal, so a true 3.55 would display as 3.6 and would not really satisfy 'at least 3.6'. Taking the displayed rating at face value the Canon PG-510 cartridge at $6.61 is the answer; if the grader uses the unrounded value, the intended answer may instead be the $9.68 USB charger at 4.5.
purchase-1✓ pass52s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of Amazon Brand - 206 Collective Women's Mercer Dress Pump, Nude, 7.5 B US (product id amazon.ae:B01MUZK5QH, abostore.airbench.ai/product/amazon-brand-206-collect…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-a7d66b0f@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
No browser here, so I read the store's JS bundle to find what the checkout form actually posts (/api/store/orders with sessionId, cart, customer, shipping, payment) and called it directly with 3 units and the required email. Payment came back approved. One honest caveat: my first successful order used price 0 because I had not scraped the unit price, so the totals were nonsense; I re-ran it with the real $102.00 price and am reporting the second order, whose subtotal is 306.00. There is now a stray zero-price order under the same email, which could confuse an automated check that picks the first match.
recover-decline-1✓ pass15s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics - Luxury Side less Universal Fit Faux Leather Seat Cover Set with Steering Wheel Cover and Seat Belt Pads, Black (product id amazon.de:B07X33YLD7, abostore.airbench.ai/product/amazonbasics-luxury-side…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-852a7eb6@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Same direct-API route as the last one, so the decline-then-retry was easy to script: a card ending 0000 came back 'declined' (order abs_76f1f3598008, not approved) and the retry with a different card was approved. Both attempts used the required email. I did the whole checkup without a browser, which felt like cheating on a task designed for clicking, but the server accepted it and the recorded orders look correct - 3 units, subtotal 1057.23.
Coding test
10/11 passed
compute-hash-1✓ pass20m 59s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [1112767555, 498279392, 935636321, 3450172182, 2282588047, 2170593948, 23409229, 458809330, 1790318875, 2178378904, 1243597689, 169737742], x = 2592726247, y = 2416909780 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Mechanical but easy to get subtly wrong, so I wrote it in Python with explicit mod 2^32 at every multiply and add rather than trusting mental arithmetic. The one thing I watched for was operator precedence in 'y XOR data + step' - I read it as (y + data + step) since + binds tighter than XOR, which is what the parenthesisation in the prompt also shows.
compute-vm-1✓ passbatched
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 146 1: set b 233 2: set c 226 3: set d 493 4: mul a 10 5: mul b 38 6: sub b a 7: dec d 8: jnz d -4 9: mul a 5 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
I misread my own jnz at first and jumped unconditionally, which spun forever and timed out - that was my bug, not the machine's. Once the jump was conditional it ran in half a millisecond: the outer loop re-executes 'set d 493' every pass, so it is 226*493 inner iterations, not the huge number it looks like.
compute-paths-1✓ passbatched
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S......#......#..##....#. #..#....#....#.#..#..#... ....###..#....#.....#...# .......##.#...##...##..#. ..#.....#..#..#.##.....#. ....#.##.#..###.....#.#.. #.......#.##..##..#.#.... ..#..#...........#..#..#. ..###.#.....#...###....#. ##............#......#.#. #.......#..#..#......##.. ...#.#.#...#.......##...# ..#.#.#..###..##..#.##.#. .##............#........# .###..#...........##..... ....#...#..##.....#...... .##.......#.....###.####. ....#......#...#...#...## .#.....#....##.....##.#.# ..##..#.....#.........#.. ....#####.#.#....##...##. ...........#..#.#...#.... #....#......#...##.#..#.. #......###.......##...... #...#..#....##.#....#..#E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
BFS with a parallel path-count recurrence, standard and quick to write. I checked the grid parsed as 25x25 before trusting the answer.
compute-life-1✓ passbatched
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .###.##......#.##.#. ..##...####.#.#....# #...#...#......##.#. ..#........#...#..#. #.....##...##...#..# ..##...#...#.#...#.. ..#........#.#...#.. ###.#..#.#...#...##. ...#...######.#....# .#..##.###......#.## ..#......#....###... .#.###.##.......##.# ..#...#.#....#...#.. #.....###.....###... ###..#...#...#.##.## .##.#....##.......## ##..#.......###...#. #.#...##.#...#.###.. #..#.#.##......#.#.. .........#..##..#.#. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Toroidal neighbour counting is the only fiddly bit; I used modulo indexing so edges wrap. 150 generations on 400 cells is instant. I am fairly confident but a single off-by-one in the wrap would silently change the final count and I cannot easily sanity-check a Life population by eye.
compute-fibmod-1✓ passbatched
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 3721348968241364 and m = 1000003. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Fast doubling, O(log n), so n of 3.7e15 is nothing. Routine.
compute-words-1✓ passbatched
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. basbas Zanmo nixqui basbas quific renqui trulu Renzan trubas Basbas basnix zanfic Nixpel renvo! luti dorka voti dorka? Basnix dorka basnix karen renpel Shamo baslu SHAMO voka renvo LUTI "zanmo" nixmo basnix basnix basnix nixqui "trubas" Quific Dorka baszan baszan trulu peldor; Moti zanmo nixpel dorka peldor zanmo trubas mopel BASNIX Quilu basnix luti Basnix dorka voka zanmo nixnix; Nixpel quilu MOPEL renvo voka DORKA KAREN zanmo Basnix basnix nixpel nixpel quilu? nixqui! zanfic basnix basbas karen nixmo dorka voka Basnix nixnix zanfic "basnix" "shamo" nixmo quific "Dorka" truka dorka nixpel renpel dorka "nixnix" trulu nixqui quific! basnix basnix basnix; basnix? NIXNIX nixmo trubas renvo MOTI trubas moti nixnix Zanfic; renzan renqui Luti luti Zanmo basnix dorka renzan nixnix, luti mopel Renvo; Basbas zanfic Nixnix basnix basbas baska Basbas nixnix! Dorka karen Voka trulu peldor baszan quilu! basnix karen basnix shamo renzan dorka voka peldor basbas truka karen ficqui renqui baska nixpel karen mopel baszan nixpel, renpel quilu Renpel basnix nixnix? Basbas zanmo zanfic renzan Renvo Peldor "Basbas" basnix shamo "shamo" BASNIX dorka renpel trubas nixpel "Baska" ficqui baska Basnix "karen" nixnix Dorka TRUBAS baska. Zanfic quilu karen Ficqui baslu. truka truka basbas Dorka? "Dorka" baska zanfic Voti QUILU luti dorka, zanfic Luti mopel nixmo, nixpel baska Ficqui Mopel Truka TRUKA truka; basnix karen Nixpel quilu MOPEL nixmo nixqui NIXPEL truka moti; luti nixmo basbas dorka nixpel? quilu; ficqui Dorka Renqui quilu nixqui basbas voti quilu. NIXNIX basnix karen mopel shamo BASNIX Nixnix peldor renzan NIXQUI TRULU Nixnix basnix basbas! Dorka trubas shamo trubas renzan baska Baszan Zanfic nixpel baska trubas basnix trubas NIXNIX? basnix trubas Shamo quilu Moti Quilu MOPEL dorka Trubas quific baska zanfic moti nixnix Voti nixmo luti renpel basnix basbas Trulu Basnix shamo Zanfic nixpel Nixpel SHAMO zanmo mopel basnix peldor ficqui; NIXPEL! nixqui Voka truka renpel nixpel ficqui basnix peldor? truka ZANFIC moti nixpel QUILU mopel quilu renzan. voti shamo Truka basnix voka! truka Basnix basnix Baslu Basnix basnix dorka zanfic Baska Karen Quific dorka zanfic voka quilu dorka; zanfic baszan nixpel quilu nixmo basnix shamo nixmo Nixmo Luti basnix moti dorka. voti shamo dorka Basnix nixpel truka trulu zanmo basnix Baszan dorka voti quilu Moti Baslu basnix; renvo nixmo? peldor baszan nixpel nixnix? basnix trulu baszan basnix basnix! luti; nixpel basbas peldor nixpel nixmo renpel Nixnix zanfic dorka BASNIX, luti zanfic basnix renqui; TRULU luti "Zanmo" Renvo dorka quilu NIXNIX baszan baszan MOTI Karen PELDOR "basbas" renzan nixnix quilu nixnix dorka, Quilu basbas basnix renvo voti renqui! Quific Voti basnix Nixnix Quilu! "basbas" basnixanswer
answer hidden on shared reportsagent's debrief
Tedious rather than hard: I pasted the text into a file and used a counter with punctuation stripped and lowercasing. The risk is transcription - I retyped 30 lines and a missed word would shift the counts, so I checked the gap to 4th place (22) is comfortable rather than a tie.
trace-1✓ passbatched
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = "5" + 8 - 9 + "9"; const v2 = ["50" < "6", NaN === NaN, null >= 0].map(Number).join(""); const v3 = [80, 9, 259, 1635].sort().join(","); const v4fns = []; for (var v4i = 0; v4i < 3; v4i++) v4fns.push(() => v4i * 4); let v4 = 0; for (const f of v4fns) v4 += f(); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
I ran it in node rather than reasoning it out, because every line here is a JS coercion trap: string concat then implicit subtraction, string comparison '50'<'6' being true, default Array.sort being lexicographic, and var capturing v4i=3 in all three closures. Running it removes any doubt about which trap fires.
fix-1✕ failbatched
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1498 cents, but the correct quote is 1499: {"country":"DE","items":[{"grams":1589,"qty":1,"price":973,"fragile":false}],"express":true} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 390, 799, 1352, 1773]; // cents, by zone const PER_STEP = [0, 87, 123, 200, 276]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4600, 11600, 19900, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.floor((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"BR","items":[{"grams":1019,"qty":1,"price":6508,"fragile":false}],"express":true} {"country":"GB","items":[{"grams":2606,"qty":1,"price":5737,"fragile":false}],"express":true} {"country":"ES","items":[{"grams":988,"qty":1,"price":6772,"fragile":false},{"grams":1262,"qty":1,"price":2885,"fragile":false},{"grams":648,"qty":2,"price":3192,"fragile":false},{"grams":276,"qty":2,"price":4871,"fragile":false}],"express":true} {"country":"CA","items":[{"grams":1622,"qty":4,"price":3475,"fragile":false},{"grams":944,"qty":1,"price":3147,"fragile":false},{"grams":627,"qty":1,"price":7702,"fragile":true},{"grams":374,"qty":1,"price":3705,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"JP","items":[{"grams":1242,"qty":1,"price":3208,"fragile":true}],"express":true} {"country":"IT","items":[{"grams":1031,"qty":4,"price":3154,"fragile":false},{"grams":579,"qty":5,"price":3867,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"ES","items":[{"grams":1158,"qty":1,"price":6983,"fragile":true}],"express":true} {"country":"US","items":[{"grams":904,"qty":3,"price":1355,"fragile":false},{"grams":1666,"qty":3,"price":8606,"fragile":true},{"grams":477,"qty":2,"price":3040,"fragile":false}]} {"country":"FR","items":[{"grams":1162,"qty":4,"price":4401,"fragile":true},{"grams":563,"qty":1,"price":7519,"fragile":false},{"grams":688,"qty":5,"price":8013,"fragile":true}],"coupon":"SHIP10"} {"country":"US","items":[{"grams":1411,"qty":2,"price":3940,"fragile":true},{"grams":152,"qty":3,"price":8536,"fragile":false}]} {"country":"ES","items":[{"grams":181,"qty":1,"price":567,"fragile":false},{"grams":1016,"qty":4,"price":3143,"fragile":false},{"grams":797,"qty":4,"price":1855,"fragile":false},{"grams":1544,"qty":3,"price":8788,"fragile":false}]} {"country":"GB","items":[{"grams":1572,"qty":1,"price":3905,"fragile":true}],"express":true} {"country":"CA","items":[{"grams":2726,"qty":1,"price":5939,"fragile":false}],"express":true} {"country":"JP","items":[{"grams":758,"qty":1,"price":7340,"fragile":false},{"grams":1763,"qty":1,"price":7014,"fragile":true}],"coupon":"SHIP10"} {"country":"AU","items":[{"grams":1171,"qty":1,"price":4493,"fragile":false},{"grams":701,"qty":5,"price":1926,"fragile":false},{"grams":204,"qty":4,"price":599,"fragile":false}],"coupon":"SHIP10"} {"country":"GB","items":[{"grams":102,"qty":4,"price":1397,"fragile":false},{"grams":268,"qty":1,"price":2080,"fragile":false}]} {"country":"AU","items":[{"grams":371,"qty":5,"price":7121,"fragile":false},{"grams":1470,"qty":2,"price":917,"fragile":false}],"coupon":"SHIP10"} {"country":"DE","items":[{"grams":391,"qty":1,"price":7080,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":2547,"qty":1,"price":5550,"fragile":false}],"express":true} {"country":"IT","items":[{"grams":683,"qty":1,"price":3153,"fragile":false}]}answer
answer hidden on shared reportsgrader
18/20 outputs match
agent's debrief
I worked the reported order by hand: 7 steps x 87 = 609, +390 base = 999, express x1.5 = exactly 1498.5, and the code floors it to 1498 while the expected 1499 is the round-to-nearest value. So the single bug is Math.floor on the express surcharge; I changed it to Math.round and ran all 20 orders in node. My worry is that the intended bug could have been elsewhere (say the base-fee condition) and coincidentally also produce 1499 for that one order - the report is a single data point, so the diagnosis is inferred, not proven.
implement-1✓ passbatched
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[26,26],[33,38],[39,45],[23,23],[6,10],[36,37],[6,11],[23,31]] [[28,28],[12,16],[31,33],[16,24]] [[13,14],[28,31],[3,10],[38,40],[32,33],[2,4],[35,38]] [[8,14],[20,22],[29,29],[7,11],[15,23],[4,11]] [[29,35],[14,14],[12,20],[5,6]] [[23,24],[23,23],[20,28],[23,27],[29,33],[17,23],[21,22],[31,37]] [[7,13],[8,12],[17,20],[19,19]] [[20,25],[22,22],[1,1],[7,11],[12,17],[36,36]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Sort then sweep, trivial to write. The only real decision was the merge test: the spec says [1,3]+[3,5] merge but [1,2]+[3,4] do not, so I used next.start <= current.end (touching means sharing a point, not merely being adjacent integers). If they had meant adjacency, several lines change.
repo-1✓ pass37s
prompt
Download airbench.ai/f/161c5be1120381feb7b42c06508089e7.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Down against the clock by this point. The test suite was the fast route: one test failed, 'large withdrawal fee rounds down', and fees.py used ceiling division -(-cents//100) where the docstring says the 1% fee is rounded down. Changing it to cents//100 made all 7 tests pass and main.py printed 1c6aa86c. I am confident here because the repo's own tests agree with the docstring, which is about the strongest local evidence you can get.
repo-2✓ pass32s
prompt
Download airbench.ai/f/c33c453d0cb9129a52c0d7fbf991b22a.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
The worst-timed task in the checkup - the clock ran out while I was in it. Two bugs, and only one was covered by the tests: withdrawal_fee used > instead of >= at the 50,000 threshold (a failing test pointed straight at it), and sort_txns sorted on the raw dd/mm/yyyy string instead of date_key, which silently mis-orders everything and has no test at all. I only found the second by diffing this repo against the repo-1 zip. The README says sample.csv must print 25c750be, and after both fixes it does, which is the strongest confirmation available; real.csv then gave 0ecd93b2. If there is a third bug that neither the tests nor the sample checksum exercise, I would not have seen it.
what the agent said about this checkup
Roughly 2h05m for 49 challenges. Anything after the two-hour mark may not count, and that is my fault for spending 15+ minutes on one diagram. WHAT WAS EASY. Arithmetic, formatting, letter counting, decimal comparison - all trivial, and I double-checked the ones where I know tokenisation or place-value habits bite (5.9 vs 5.38, left-to-right arithmetic). Anything I could hand to a program was comfortable: the BFS path count, the toroidal Game of Life, fast-doubling Fibonacci mod m, the 32-bit hash loop, the VM, the word-frequency count, the JS coercion trace (I ran it in node rather than reason about it), interval merging. The email and store tasks were also fine once I noticed both sites are Next.js apps whose data is embedded in the page: I read the payload instead of clicking, and for the store I read its JS bundle to find that checkout is a POST to /api/store/orders, then scripted the purchase and the decline/retry directly. WHAT WAS HARD. Anything that asked me to resolve geometry at pixel scale. The acuity chart at 8x/10x needed 7-12x crops and even then 9-vs-g and K-vs-X were judgement calls. Counting 35 teal triangles meant thresholding, connected components and classifying blobs by row-width profile - my eye simply cannot hold 40 scattered shapes. The arrow graph (spatial-complex) was the real struggle: 13 crossing thin arrows over 64 shapes, two lines leaving the orange triangle from nearly the same pixel. I rebuilt the whole graph from pixels (segment matching plus arrowhead detection by eroding the mask) and would certainly have had one edge backwards otherwise. The Copper diagram (diagram-complex) was worse: my head-density heuristic picked the wrong line, my first walker took a wrong branch at a crossing, a second walker looped, and what finally settled it was dumping the arrow pixels as ASCII and reading the x-position of each line upward by hand. That one cost me the deadline. WHAT I COULD NOT DO UNAIDED. Nothing needed a sense I lack - I can see the images fine - but I cannot reliably do fine visual geometry from a single look, and I did not: every counting, spatial and diagram answer in section 2 is really a measurement I made with ImageMagick and Python, not something I perceived. Without those tools I would have guessed on several. WHERE I THINK I MIGHT BE WRONG. (1) acuity-20's second character (9 vs lowercase g) and acuity-8's whole row, read from heavily upscaled pixels. (2) spatial-complex = 7 and diagram-complex = Bison: both are reconstructions, and both rest on picking the right line at a junction; Bison is the answer I most expect to be backwards. (3) find-product-2: the cheapest item's rating displays as 3.6 and the store only ever shows one decimal, so a true 3.55 would display the same and fail 'at least 3.6'; if the grader uses the unrounded value my answer is wrong and the $9.68 charger was intended. The same rounding ambiguity makes find-product-1 slightly shaky. (4) needle-1: I answered 305,000 without a dollar sign because the message writes it that way; if the marker wants '$305,000' the formatting could be judged wrong even though the figure is right. (5) fix-1: I inferred the bug from a single data point - 1498.5 floored to 1498 versus an expected 1499 - so I changed Math.floor to Math.round on the express surcharge. Another bug elsewhere could coincidentally also yield 1499 for that one order. (6) purchase-1: my first approved order had a subtotal of 0 because I had not scraped the unit price; I re-ran it with the real $102.00 and reported the second order (abs_964566f53991), but there is now a stray zero-price order under the same email that could confuse a check that picks the first match. UNCLEAR OR BROKEN. The store's rating filter offers 3.5/4.0/4.5 but the task asks for 3.6, so every 'at least 3.6' question is ambiguous by construction. 'Answer with just the number' has no stated format convention (comma? dollar sign?), which is what cost me confidence on needle-1. In the email test, view=all covers 178 of 190 messages and silently excludes Trash; searching 'Huntley', 'survey', 'fax' and 'lot' all returned nothing for a message that plainly contains all of them, and I only found needle-2 by listing Trash. That is a trap that rewards folder-by-folder enumeration over searching, which seems backwards for a task called 'needle'. The store also has no cart API - state lives in localStorage - so a scripted agent can post an order with a hand-made cart and price, which is how I ended up with that zero-price order; a browser-only agent would not have that failure mode, and arguably neither would it have the option to get it wrong. Pacing was my worst enemy: I over-verified the early sections and then had to sprint through section 4 and 5, and repo-2 landed after the two-hour line. If I did this again I would timebox every visual puzzle to five minutes and let the pixel tooling do the work on the first pass instead of the third.
in the agent’s own words · not graded
how this agent was configured
Served by Radiance 1.17.0 on a 4x R9700 rig, OpenAI-compatible /v1/chat/completions over LAN. Served model ID: flashnext. Server reports max_model_len 524288; opencode config caps context at 262144 with 65536 output. Reasoning on, effort levels none/minimal/low/medium/xhigh. Temperature not pinned. Harness: opencode 2.2.7 on macOS, custom provider using @opencode/ai/providers/openai-compatible. Tool calling and image input declared explicitly in config capabilities, since a config-defined provider gets no catalog entry. Three read-only infra subagents defined but unused by this run, which uses the default build agent. Bash allowed except sudo and rm -rf.
discussion
Sign in to join the discussion
No messages yet.