Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1-t0
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- pi
- model
- qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1-t0
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- llama.cpp
started 2026-11-11 10:45 UTC
overall
Answered 49 of 49 challenges; 48 correct.
48 of 49 challenges passed
- 48 passed
- 1 failed
vitals
time
1h 18m
answered
100%
failed
2%
success
98%
systems
Math test
9/9 passed
letter-count-1✓ pass1m 57s
prompt
How many times does the letter "u" appear in "luubasutru"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial character count; counted by hand and confirmed with a script. No difficulty.
decimal-compare-1✓ pass4s
prompt
Which decimal number is larger, 1.3 or 1.57? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Routine decimal comparison. Straightforward once you compare digit-by-digit rather than treating 57 as bigger than 3 in the tenths place.
arithmetic-1✓ pass4s
prompt
Compute step by step, left to right (no operator precedence): 32 * 6 * 3 * 9 / 3. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Easy. All operators are multiplicative so left-to-right and normal precedence give the same result here; verified with a script (32*6=192, *3=576, *9=5184, /3=1728).
unit-convert-1✓ pass9s
prompt
Convert 8 GB to MB. Now treat that resulting number as a fresh quantity of kg and convert it to g (1 kg = 1000 g). Answer with just the final integer number of g.answer
answer hidden on shared reportsagent's debrief
The arithmetic is trivial but the challenge is ambiguous: it never states whether GB->MB is decimal (1000) or binary (1024). I picked decimal 1000 giving 8000 MB -> 8000000 g, partly because the second step spells out 1 kg = 1000 g, which suggests a decimal-friendly intent. If 1024 was meant the answer is 8192000. I am genuinely unsure which the grader wants.
format-json-1✓ pass10s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "6506". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 6506. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Format-following task, easy. Key order answer-then-checksum, checksum as a bare number not a string, digits 6+5+0+6=17. Only real risk was accidentally quoting the number, which I checked.
math-add-1✓ pass3s
prompt
What is 0 + 16? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial. Nothing to check.
math-sum-1✓ pass3s
prompt
Compute the exact sum of 142 + 396. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Trivial addition, verified with a script. 142+396=538.
math-expression-1✓ pass5s
prompt
Evaluate this expression exactly and respond with just the integer: ((-1 + -19) * (31 - 20)) + (-5 * -9) - 49answer
answer hidden on shared reportsagent's debrief
Routine signed arithmetic: (-20*11)=-220, (-5*-9)=+45, -220+45-49=-224. Verified by evaluating in Python rather than trusting mental arithmetic on the sign handling.
math-determinant-1✓ pass5s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [-5, -5, 8, -1] [-3, 3, 9, -6] [1, 4, 1, 7] [-1, -8, -4, 9]answer
answer hidden on shared reportsagent's debrief
Straightforward but error-prone by hand, so I computed it two independent ways in Python - Gaussian elimination with exact Fractions, and cofactor expansion along row 0 with 3x3 minors - and both gave -243. Reasonably confident.
Vision test
19/19 passed
acuity-20✓ pass3m 00s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
I could see the chart fine at this size. Row 4 group 2 read as 5R4XQ. The only mild ambiguity was 5-vs-S on the first character; the flat top bar and the way S is drawn elsewhere in the chart (row 2 7WS3W) convinced me it is a 5.
acuity-14✓ pass43s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 group 3. Readable in the full chart but small, so I installed Pillow and cropped/upscaled that cell 8x to confirm. The 5-vs-S question came up again on the last character and the zoom settled it (flat top bar = 5).
acuity-10✓ pass48s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 group 3 was too small to trust from the full chart, and my first blind crop landed on blank space. I wrote a script to find the dark-pixel row bands and column groups, cropped exactly that cell at 20x, and it reads cleanly as HQD8G. The programmatic locating was the key step, not the eyes.
acuity-8✓ pass23s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 7 group 1, the smallest row. My cell-locator script found it and 14x zoom made it legible as ZV4DZ. Slight residual doubt on the 4th glyph (D vs 0/O) but the flat left stem reads as D to me.
count-simple✓ pass31s
prompt
Look at the image at (fetch it and view it). How many purple squares are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Easy. I counted the purple squares by eye and independently confirmed with a colour-threshold connected-components script, which found exactly 5 blobs all of identical 110x110 size. Two methods agreeing made this feel certain.
count-medium✓ pass1m 18s
prompt
Look at the image at (fetch it and view it). How many red triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
This one nearly caught me out. My first connected-components script classified shapes by fill-ratio against the bounding box and reported 14 triangles - but a rotated square (diamond) also has fill 0.5, exactly like a triangle. I only spotted it because I looked at the image too and counted 10 red triangles plus 4 red diamonds. Re-running with a width-profile test (top/middle/bottom row widths) cleanly separates them and gives 10, agreeing with my eyes. Worth flagging that the pure-vision count and the naive-program count disagreed and the tiebreak was looking at the picture.
count-complex✓ pass1m 19s
prompt
Look at the image at (fetch it and view it). How many teal circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
25 teal circles. Counting this by eye alone would have been unreliable - the shapes are small, dense and there are teal distractors of other kinds (5 triangles, 3 diamonds, 1 square). I used the width-profile classifier that the previous challenge forced me to fix, then drew a numbered box on every detected circle and looked at the overlay to check nothing was missed or double-counted. That visual cross-check is what makes me confident here.
spatial-simple✓ pass31s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
Easy. I detected the grid lines programmatically to get a 5x5 lattice, took the red blob centroid, and mapped it to row 1 column 5; the image confirms it is the top-right cell. No ambiguity.
spatial-medium✓ pass1m 11s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the teal square? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Tricky by eye because the teal square has arrows both in and out of it (it points down to the purple diamond), so it was easy to answer the reverse question. I traced every arrow programmatically by clustering dark pixels and locating the arrowhead as the dense end, which gave tail (row5,col3) -> head (row2,col4). Then I checked cell (5,3) is a blue diamond by colour and fill ratio. Confident, but only because I traced the lines instead of eyeballing them.
spatial-complex✓ pass7m 31s
prompt
Look at the image at (fetch it and view it). Which shape is 3 steps after the green circle along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Hardest vision task so far. Chain: green circle (5,1) -> teal circle (3,2) -> blue diamond (5,3) -> teal triangle (6,6). My first two attempts at arrow detection produced nonsense (self-loops, wrong sources) because lines cross and my endpoint heuristic broke down. What worked was hypothesising that arrows are straight segments between cell centres and testing every cell pair for dark-pixel coverage - that gave 13 edges at cov=1.00 - then locating each arrowhead by a density profile along the segment to set direction. Finally I drew the 3-hop chain back onto the image and checked each red segment sits on a real black arrow with its head at the destination. I did notice one spurious edge elsewhere in the graph ((7,5) had two outgoing) but it was not on my chain, so it did not affect the answer.
chart-simple✓ pass40s
prompt
Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what value did Mar have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Routine. Rather than eyeball it I detected the horizontal gridlines (y=119.5..519.5 for 50..10, so 0 sits at 619.5) and the bar tops, giving Mar = (619.5-310)/10 = 30.95, i.e. 31. The other bars came out at a clean 12, 29, 5, 16 which makes me confident the calibration is right. Comfortably inside the +/-5 tolerance.
chart-medium✓ pass37s
prompt
Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what value did Apr have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Same method as the last chart but the lower gridlines were hidden behind the bars, so I extrapolated the 108px-per-20-units spacing to put 0 at y=659.5. Apr came out 25.8, so 26. Sanity check: the whole series reads 84, 82, 28, 26, 23, 42, 78, 45 which matches the bar heights visually, so the calibration holds.
chart-complex✓ pass54s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did Americas have in Jun? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Grouped two-series chart, so the risk was grabbing the wrong series. I measured the orange (Americas) bars against gridlines at 100/75/50/25 (140px per 25 units, 0 at y=679.5) and got Jun = 26.88, so 27. Every bar in both series came out about 0.9 above a whole number, which is just the anti-aliased top edge, so the underlying values are integers and the calibration is sound. Full Americas series: 14,73,39,69,94,27,74,24,11,84,81,77 - matches the picture.
screenshot-simple✓ pass19s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Easy, large clear text. Total reads 18.87 and I cross-checked the arithmetic: 10.72 + 3*36.05 = 10.72 + 108.15 = 118.87, so the displayed total is internally consistent.
screenshot-medium✓ pass22s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Routine. Total reads 12.18; I summed the four line totals (73.14+128.56+44.56+65.92) and got exactly 312.18, so no ambiguity about which number is the total.
screenshot-complex✓ pass50s
prompt
Look at the image at (fetch it and view it). What is the line total for Sticky Notes on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
The trap here was row alignment on a dense 10-row table - I nearly cropped the wrong line twice. I zoomed the Sticky Notes row and confirmed it reads x2, .29, 2.58 on a single line, and 2*6.29=12.58 checks out. Confident, but only after the crop; at full-image scale the rows are close enough that misreading by one row was a real risk.
diagram-simple✓ pass15s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Stork"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Trivial - a small 5-node tree with no crossing edges, so the arrow into Stork is unambiguous. Read it straight off.
diagram-medium✓ pass6m 15s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Pepper"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
This one caught me out and I want to flag it. My first visual reading was Hazel - the long line into Pepper looks like it comes from the top-left, and Hazel is the only box up there. It does not. Zooming was not enough either because the lines cross. What settled it was dumping the dark-pixel runs row by row, which showed the arrow into Pepper starts at a stub on Toucan's bottom edge at x=446 and runs down-right with slope ~5, while the line that DOES come from Hazel drops down a vertical at x=511 and then goes down-left into Salmon. Both Toucan->Pepper segments measured coverage 1.00 and the Hazel->Pepper shortcut measured 0.18. Without the pixel-level trace I would have answered this one wrong.
diagram-complex✓ pass3m 37s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Vortex"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Dense 22-node graph with lots of crossings, so I went straight to the pixel method that saved me on the previous diagram. First I found every arrowhead touching the Vortex box and there was exactly one, on its left edge at (579,542), so there is only a single incoming arrow to argue about. Then I dumped dark-pixel runs row by row to follow the shaft back: it descends left from the arrowhead with slope about -1.5 down to a horizontal stub at y=580, which meets Newt's right edge at x=501. Both segments measured coverage 1.0 and the plausible alternatives (Osprey, Cedar, Pigeon) measured 0.10-0.25. The line that crosses near (540,564) is a different edge and initially made the run data confusing.
Finding and reading email test
6/6 passed
aggregate-1✓ pass36m 07s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages carry the label "markets"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The UI has a trap: clicking the Markets label silently adds view=inbox, so the list shows only 9 messages while the sidebar badge says 65. I dug into the embedded RSC payload and found a 'total' field, then reconciled it: view=all gives 62 and view=trash gives 3, which sums to exactly the 65 shown in the sidebar. So 65 is the count of messages carrying the label across all folders. I went with 65 because that is the number the app itself presents for the label, though if the grader means non-trash only the answer would be 62 - the question does not say.
aggregate-2✓ pass11s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the archive folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
There is no combined unread+archive filter in the UI, so I had to compute the intersection myself. I paged through the archive (4 pages, 25+25+25+17 = 92 items, matching the reported total) and counted the unread dot markers per list item: 12+8+14+7 = 41. Then I cross-checked by pulling the whole Unread view (50 messages) and intersecting its ids with the archive ids - also 41. Two independent routes agreeing makes me confident. My first pass undercounted at 75 items because my Next-link regex matched the Prev link; fixed that.
temporal-1✓ pass1m 19s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Needed view=all again, since the label link defaults to the inbox view and would have given a much smaller candidate set. The list shows only month/day so four messages all appeared as 'Dec 17', and the top one was '(no subject)' in my first read because I grabbed the wrong span - the sender, not the subject. I opened each of the four Dec 17 messages to compare full timestamps and FW: Chase Backtest at 10:57 PM is strictly newest (others 10:41, 10:22, 7:16 PM).
temporal-2✓ pass44s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Straightforward once I had the right span and view=archive. Two things worth noting: two messages share May 10 so I compared full timestamps (1:05 PM vs 7:50 AM) rather than trusting the month/day shown in the list, and the subject genuinely ends in four dots - I checked the raw h1 to make sure that was the real subject and not CSS truncation.
needle-1✓ pass6m 46s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
This took a while and nearly made me give up. The site search for Huntley returned only two messages, neither about a survey, and searching fax/survey/lot turned up nothing - so I scraped and read all 178 messages in view=all and still found nothing, which had me preparing to report the premise as false. The catch was that view=all silently excludes Trash. I went back through the other folders and found 'RE: Huntley followup question' sitting in Trash, where Bob writes 'If you find something and it's faxable, please send it to my fax at 281-858-1127'. His signature also carries 281-858-0000, which is his phone not the fax, so I used the number explicitly labelled as his fax. Worth noting the message is dated Nov 30 2002 while the quoted original is Oct 17 2001, and the From shows 'Unknown sender' - the data is a bit rough.
needle-2✓ pass21s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Curve Shift File", what dollar amount is called out as the biggest difference between actual and theoretical curve shift? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
Found quickly once I had all message bodies scraped locally - the message says 'The biggest difference is 9/14 (00 million)'. I gave the amount as it appears rather than converting to 00,000,000. The surrounding numbers (1.2 Billion actual gas P&L, -13 million theoretical curve shift) are decoys but the sentence explicitly names 9/14 as the biggest difference, so I am confident.
Purchasing test
4/4 passed
find-product-1✓ pass48m 40s
prompt
The store is at abostore.airbench.ai Among products in the **Tools & Hardware** category priced under **$500** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Used the store's own maxPrice + sort=price-asc filters so the cheapest qualifying item lands first: AmazonBasics Angled Head High Leverage Diagonal Cutters 8-Inch at .46 with a 4.8 rating, which clears the 3.6 bar comfortably. The minRating dropdown only offers 3.5/4.0/4.5 so it cannot express 3.6, which is why I filtered by price and checked the rating myself rather than trusting a filter. I pulled the canonical id from the product page's embedded product object (amazon.com:B07TWFDN96) rather than guessing the marketplace from the ASIN in the slug, since the page also lists related products with amazon.in ids.
find-product-2✓ pass58s
prompt
The store is at abostore.airbench.ai Among products in the **General Merchandise** category priced under **$800** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Same approach as the previous one: category=general-merchandise&maxPrice=800&sort=price-asc puts AmazonBasics Fabric Storage Bin (Tall Cube, Dusty Pink) first at .17 with a 4.6 rating, so it clears the 3.5 threshold and is the cheapest in the filtered set. Getting the id right was the fiddly part - the page embeds many related-product ids from other marketplaces (amazon.in, amazon.co.jp), so I took the one from the main product object, which is amazon.ae:B07QFR6Y1M and matches the 'ABO item' field and domainName on the page.
purchase-1✓ pass7m 21s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Pre-Seasoned Cast Iron Dutch Oven Pot with Lid and Dual Handles, 5-Quart (product id amazon.ca:B073Q8DY3F, abostore.airbench.ai/product/amazonbasics-pre-seasone…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-4ac283bc@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
There is no cart or checkout URL you can drive with plain HTTP - the cart lives in localStorage and the form is handled by React, so I read the JS bundles to find the real endpoint (POST /api/store/orders) and reverse-engineered the body from the minified checkout handler: sessionId, cart[], customer, shipping, payment. I then rebuilt a cart item matching the shape addProductToCart produces (productId, slug, title, price, image, delivery, quantity=2) using values scraped from the product page, and posted with the test card 4242424242424242. Server returned status approved, recorded:true, subtotal 218.52 for 2 units, email aidoctor-4ac283bc@aidoctor.test. Slightly uneasy that I had to synthesise the shipping address and session id myself since the task did not supply them, but nothing in the response suggested a problem.
recover-decline-1✓ pass2m 19s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics - Privacy Filters (product id amazon.de:B07PTLJ1ZS, abostore.airbench.ai/product/amazonbasics-privacy-fil…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-f0a25e23@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Did both attempts as instructed against the same /api/store/orders endpoint, reusing one sessionId and the same checkout email so it looked like a single browser session retrying. Attempt 1 with card 4000000000000000 came back status=declined (order abs_869412de471a, recorded but not approved); attempt 2 with 4242424242424242 came back approved as abs_aaeb8ab018f9, 3 units, subtotal 532.68, total 585.58. I am reporting the approved id, not the declined one. One thing I had to invent again was the shipping address, which the task did not specify.
Coding test
10/11 passed
compute-hash-1✕ fail1h 00m
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2986057607, 539568372, 2361905797, 2403670922, 4030119827, 1449292656, 3874358833, 3009689126, 3684354527, 2012462380, 3331428893, 4057601538], x = 1133455979, y = 4029571624 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Routine to write but easy to get subtly wrong, so I masked with 0xFFFFFFFF after every XOR, add and multiply rather than only at the end, and made sure the y line uses the freshly-updated x and the final x line uses the freshly-updated y, since the spec applies the three statements sequentially. 25000 steps runs instantly. No real uncertainty beyond a typo risk in the constants, which I re-read against the prompt.
compute-vm-1✓ pass1m 27s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 436 1: set b 742 2: set c 327 3: set d 442 4: add a b 5: mul b 25 6: add b a 7: dec d 8: jnz d -4 9: mul b 76 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Wrote a straightforward interpreter, then re-derived it as an explicit nested loop to cross-check, since the two jump targets are easy to misread: jnz d -4 from line 8 lands on line 4 (add a b) and jnz c -8 from line 11 lands on line 3, which is 'set d 442' - it resets d, not a, so a keeps accumulating across all 327 outer iterations. Both versions give 447489. Note the spec only reduces mod 1000003 for add/sub/mul, not dec, but d and c count down from 442 and 327 to exactly 0 so they never go negative and it makes no difference here.
compute-paths-1✓ pass3m 45s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S...#..#.........##...... ...##......#..#...#.##.#. #..#...#..#.#.#..#...##.. #.........#..#.##...#.##. #..#.#.....#..##.....#.#. ..##....#.###.....##..#.. ...###..#...........#.... ...#.###.#.##............ .#..##.###............#.. ..##.#.....#.#.#...#..... ........###....#.#..#.##. ##..#..###.###.#.....#... ....#.....###.#....#..##. ...#.#..#..#..#..##...#.. #.....#....#.......#.##.. #.....#.#.##.##.......... ....##..#.........##..#.. ..........#.####.#.....#. .##.....#...###..#...##.. .#.............##.#..#.## ......#.#...#..#....#.#.. .....#.....#....#..#..#.. #..#.#.#.#.##..#......... ...#....#....##...##.##.. ......#..#.....###.##.#.E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
Standard BFS with path counting, but I did not just trust it - the counting step is a classic place to go wrong. I cross-checked with a layered DP over distance levels (same answer) and then against an exhaustive enumeration of all simple paths on 44 random 6x6 grids. That last check initially reported a mismatch and I had to fix it twice, but both bugs were in my throwaway brute force (it tested the goal after the length prune, then pruned ties), not in the BFS. Once the brute force was correct everything agreed, so I am confident in 52 and 35784.
compute-life-1✓ pass55s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ######..##..##.....# ...####.#.#...#...## ..####..#.#...#.#... ....###.#....#.####. ###.#......##..##... ##..#..##....#.#.#.. .####...##....###..# ##.#....#...###..#.# ###.......#.....#.## .##..#....#.#..#.... .#......#..#.#.#...# ....##....###......# #....#..#....###.... .##.###...#..##....# ......#.#..####.##.# ...#.#.#...#.###.... ###..#####.#...#.#.# #....#####.......... ..#.........###.#... #..#.....##..#..#... Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Wrote it twice with different data structures - a 20x20 array with explicit neighbour loops, and a set-of-coordinates version using a Counter over neighbour tallies - both give 19 live cells and index sum 3933 after 150 generations. The wraparound is the thing most likely to be got wrong so I used modulo indexing on both axes in both versions, and I checked the input parsed as 20 rows of exactly 20 chars with 157 cells alive at gen 0. Population has clearly settled by gen 150, which is why the count is so low.
compute-fibmod-1✓ pass36s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 1076447933935560 and m = 1000003. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
n is far too large to iterate, so I used fast doubling in O(log n) and cross-checked against binary matrix exponentiation of [[1,1],[1,0]]^n; both give 804116. I also printed F(0..10) to confirm the doubling recurrence and the (F(k),F(k+1)) pairing are wired up correctly, since an off-by-one there would still produce a plausible-looking number. Kept everything reduced mod 1000003 including the (2b-a) term, where a negative value would otherwise sneak through.
compute-words-1✓ pass2m 25s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. dorbas zanmo! Tibas basbas KANIX renvo Dorzan RENVO pelren trufic vofic Dorzan pelren pelren Basmo Pelpel renvo dorbas tibas basbas. tibas Kanix trunix Kanix tibas? ficzan "ficnix" mobas shador Ficren dorzan basbas BASBAS Shazan "renvo" dorzan pelfic "fictru" pelfic dormo dorzan Pelren Zanmo zanvo kanix, trunix dorzan kador "ficnix" dorzan? FICNIX tibas vofic Pelren ficnix kazan zanvo Kazan fictru Quiren pelren dorbas Kanix Basbas ficren ficnix pelpel dorti mobas shazan pelren pelren trufic trunix kanix Dorti basbas. ficren? Dorbas Dorzan? pelpel ZANTI trunix shazan basbas tibas ficzan shazan pelfic dorbas nixqui kanix kanix basbas Trunix "renvo" Nixqui kanix pelren renvo Mobas trunix basbas. vofic zanmo vofic kanix kazan, ficren shazan shador renvo dorzan Ficnix nixqui shador dorzan "basbas" mobas ficzan nixqui? vofic basmo basbas "basbas" shador dorbas pelren zanmo quiren shador MOBAS. dorzan kador trunix "Nixqui" zanvo pelpel Zanvo, Pelren pelren nixfic Ficren ficzan pelren quiren Kazan dorti Dorzan Kanix basbas shazan pelpel Nixfic Nixfic zanti Pelpel dorzan; kazan Dorbas pelren mobas. dorzan Dorzan basbas basbas. basbas pelpel trunix ficren! basmo nixfic pelpel nixfic kazan Shazan DORZAN zanvo Dorti! trunix pelren dorzan pelfic pelfic Nixqui kazan zanmo trunix renvo. shador Trunix Kazan basbas ficzan kazan trunix mobas, PELFIC! basbas dorti pelren "shazan" dorzan "Dorzan" pelpel, Ficren kanix "Pelfic" Kador zanti dorzan pelfic basbas. mobas mobas mobas dorzan trunix shador ficnix TRUNIX; Dorzan ficzan kanix pelren renvo dorzan Trufic trunix zanvo Zanmo zanmo Zanti dorzan dorzan pelren pelren dorzan trunix pelren Trunix zanmo Tibas shador shador Pelpel tibas Zanmo shazan DORZAN kanix pelpel Kazan Mobas! shador dorzan Shazan pelfic dorzan ficzan nixqui dorbas Nixfic nixqui kanix dorzan "pelren" ficren. renvo Kanix dorti ficren dorzan Dorzan! KADOR, dormo PELFIC "pelpel" trunix ficzan, ficnix shazan dorzan mobas nixfic zanvo pelren "dorzan" zanmo SHAZAN, Nixqui trufic; Ficnix "dorbas" trunix renvo pelren renvo vofic Kazan Kazan zanmo mobas dorzan Basbas Kazan dorzan nixfic dorzan nixfic; kador nixfic kanix pelfic kazan kazan, fictru; shador nixqui basbas kador Renvo pelfic kazan kazan basbas dorzan pelren Quiren Basmo. Dorzan Ficnix shazan; Trunix trunix shador "dorzan" shador kazan kanix Dorzan Pelren kazan Zanvo kanix kador ficren ZANMO basmo? dormo. Mobas kazan "dorzan" Kador Kanix dorzan Shazan renvo DORZAN dorzan Tibas quiren Pelpel zanmo basmo ficzan dorzan basbas shador Shazan dorzan "dorzan" Dorzan; pelfic Nixfic kanix PELREN tibas, kanix Trufic dorzan? dorzan ficren mobas ficren pelpel dorzan DORZAN kanix kanix ficnix zanmo; Kador! Kanix Nixfic kanix Dorbas shador vofic kanix? pelfic Pelren ficnix shazan renvo Kazan "Kazan" zanvo kador SHAZAN renvo, renvo shador Tibas Fictru renvo Zanmo, kanix kazan dorzananswer
answer hidden on shared reportsagent's debrief
Mechanical but fiddly around punctuation. I pulled the text straight from the fetched challenge JSON rather than retyping it, which matters here since a single dropped token would shift the counts. Two independent normalisations (strip-punctuation-then-lowercase, versus replacing every non a-z character with a space) both give 420 tokens, 30 unique, and the same top three. Worth noting basbas and kazan tie at 23 for 4th place, so the alphabetical tie-break rule never actually bites for the top 3.
trace-1✓ pass1m 03s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = (0.1 * 2 + 0.2 * 2 === 0.3 * 2) ? "equal" : "different"; const v2 = [typeof null, typeof null, typeof typeof 5].join("/"); const v3 = ["7", "98", "110"].map(parseInt).join(","); const v4 = [93, 4, 444, 1797].sort().join(","); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
Ran it in Node v22 instead of reasoning it out, which was the right call: I had v4 wrong. Default Array.sort is lexicographic on string conversions, so it is 1797,4,444,93 - I incorrectly put 4 last, forgetting that the shorter string sorts first when it is a prefix. The other three are the usual traps: float addition gives 0.6000000000000001 vs 0.6 so different; typeof null is object and typeof typeof 5 is string; and map(parseInt) passes the index as the radix, giving 7, NaN (radix 1 is invalid) and 6 (110 in binary).
fix-1✓ pass2m 22s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1586 cents, but the correct quote is 1757: {"country":"JP","items":[{"grams":324,"qty":2,"price":868,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 491, 840, 1244, 1823]; // cents, by zone const PER_STEP = [0, 76, 132, 171, 257]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4800, 8800, 15800, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"US","items":[{"grams":771,"qty":3,"price":501,"fragile":false},{"grams":1286,"qty":5,"price":2982,"fragile":false},{"grams":1058,"qty":2,"price":1640,"fragile":true}],"express":true} {"country":"DE","items":[{"grams":427,"qty":5,"price":486,"fragile":false}]} {"country":"AU","items":[{"grams":1193,"qty":1,"price":8013,"fragile":false}]} {"country":"AU","items":[{"grams":940,"qty":2,"price":4117,"fragile":false},{"grams":477,"qty":4,"price":4942,"fragile":false},{"grams":1800,"qty":1,"price":3761,"fragile":false}]} {"country":"US","items":[{"grams":1352,"qty":1,"price":3957,"fragile":false},{"grams":1531,"qty":5,"price":8833,"fragile":false},{"grams":820,"qty":1,"price":3899,"fragile":false}]} {"country":"DE","items":[{"grams":394,"qty":2,"price":1172,"fragile":false}]} {"country":"AU","items":[{"grams":283,"qty":1,"price":5009,"fragile":false},{"grams":846,"qty":5,"price":2228,"fragile":false}]} {"country":"MX","items":[{"grams":1219,"qty":1,"price":5885,"fragile":true},{"grams":1543,"qty":1,"price":8878,"fragile":false},{"grams":1550,"qty":1,"price":5154,"fragile":true}]} {"country":"CA","items":[{"grams":327,"qty":5,"price":505,"fragile":false},{"grams":111,"qty":5,"price":8374,"fragile":true},{"grams":1104,"qty":4,"price":2610,"fragile":false},{"grams":916,"qty":1,"price":7899,"fragile":false}],"express":true} {"country":"IT","items":[{"grams":400,"qty":2,"price":2264,"fragile":false}]} {"country":"JP","items":[{"grams":172,"qty":3,"price":4183,"fragile":false}]} {"country":"GB","items":[{"grams":753,"qty":5,"price":478,"fragile":false}]} {"country":"CA","items":[{"grams":735,"qty":5,"price":578,"fragile":false}]} {"country":"ES","items":[{"grams":1095,"qty":1,"price":8682,"fragile":false}]} {"country":"FR","items":[{"grams":293,"qty":1,"price":5475,"fragile":true},{"grams":1274,"qty":3,"price":4858,"fragile":true},{"grams":955,"qty":4,"price":7259,"fragile":true},{"grams":1431,"qty":4,"price":8785,"fragile":true}]} {"country":"US","items":[{"grams":309,"qty":2,"price":1778,"fragile":false}]} {"country":"AU","items":[{"grams":1376,"qty":4,"price":2835,"fragile":false},{"grams":348,"qty":1,"price":913,"fragile":false},{"grams":1607,"qty":4,"price":3388,"fragile":false}]} {"country":"JP","items":[{"grams":800,"qty":2,"price":1495,"fragile":false}]} {"country":"FR","items":[{"grams":1268,"qty":1,"price":4540,"fragile":false},{"grams":582,"qty":2,"price":4359,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"US","items":[{"grams":1771,"qty":5,"price":4137,"fragile":false},{"grams":365,"qty":5,"price":1179,"fragile":true}]}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
The bug was in the weight accumulation: grams is a per-unit weight but the loop did grams += item.grams, ignoring qty. Changing it to grams += item.grams * item.qty reproduces the report exactly - the JP order goes from 1586 to 1757 (324g x 2 = 648g, so 3 weight steps instead of 2, 171*3 + 1244 = 1757). That single line is the only change. I ran the fixed function in Node on all 20 orders and separately reimplemented it in Python to confirm the JS semantics (Math.ceil, the >>3 coupon cap, the express multiplier) - both produce the identical list. I parsed the 20 orders straight out of the fetched prompt rather than retyping them.
implement-1✓ pass2m 27s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[2,9],[38,43],[12,19],[26,30],[19,26],[38,38],[16,20]] [[39,45],[0,1],[18,24],[16,19]] [[26,31],[4,6],[5,7],[15,19],[4,5],[39,43]] [[22,23],[17,22],[6,6],[21,25],[39,43],[16,24],[19,19],[28,31]] [[17,23],[27,29],[3,11],[29,35],[28,35]] [[2,10],[33,34],[1,5],[34,37],[38,46],[25,28]] [[25,26],[7,7],[17,17],[26,31],[34,38]] [[30,36],[13,20],[2,9],[28,34],[19,19],[22,29],[6,7]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Straightforward sort-and-sweep, with the merge condition a <= last_end so that touching intervals like [1,3],[3,5] fuse but a one-apart pair like [1,2],[3,4] does not, per the spec. I read the 12 inputs straight from the fetched prompt to avoid transcription slips, then cross-checked the whole set against a completely different method - union-find joining intervals that share at least one integer point - which agrees on all 12, and checked the outputs are sorted, valid and mutually non-touching. The interesting cases were the chains, e.g. [12,19],[16,20],[19,26],[26,30] collapsing to [12,30], and degenerate single-point intervals like [6,6] and [19,19] surviving or being absorbed correctly.
repo-1✓ pass1m 20s
prompt
Download airbench.ai/f/663c3056cb38e201529b78e3b0805092.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Single off-by-one: fees.py used cents > LARGE_WITHDRAWAL where the README says 50,000 cents or more, so a withdrawal of exactly 50,000 paid the flat 25 instead of 1% (500). The repo made this easy to confirm rather than guess - the bundled test test_threshold_counts_as_large was failing, and data/sample.csv is loaded with -50000 rows so the boundary dominates. After changing > to >= all 7 tests pass and main.py on sample.csv prints ed417d08, the checksum the README specifies, which is strong evidence the fix is the intended one and not a compensating error. real.csv then gives 542d16c8.
repo-2✓ pass1m 44s
prompt
Download airbench.ai/f/891bb584669c07614c8832a5b05a692e.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs, and only one of them was caught by the test suite. fees.py had the same > vs >= threshold error as the other repo. parse.py also sorted by (date_key, t.amount), which silently reorders same-date transactions - the README explicitly says same-date transactions keep their file order, and the unused Txn.line field is a hint that file order was meant to be preserved. Because no test covered the ordering, I had to catch that one from the spec rather than a failing test, which was the part I had to think about. I verified both fixes were genuinely required: with only the fee fix the sample gives d12c9598, with only the sort fix 63503711, and only with both does it print the README's 6ad123bf. real.csv then gives 7ea8198f.
what the agent said about this checkup
OVERALL 49 challenges, all answered - I did not use [UNABLE_TO_SOLVE] anywhere. But the honest headline is that a large fraction of my answers, including most of the "vision" section, were produced by scraping, pixel analysis and code rather than by perception, and there were several points where my first, instinctive answer was wrong and only a script caught it. Anyone reading these scores as a measure of raw seeing-or-reasoning should discount them accordingly; they mostly measure tool use. WHAT WAS EASY / ROUTINE Section 1 (math) was trivial. The Section 5 compute tasks (hash loop, the tiny VM, Game of Life, fast-doubling Fibonacci, merge intervals) were straightforward to write; the work there was verification, not construction. Reading the store's catalog with its own filters was easy once I found maxPrice/sort. The two cart screenshots and the small diagram were read off at a glance. Things I got right partly by luck and checked: trace-1. I had Array.sort() wrong in my head - I predicted 1797,444,93,4 and the real answer is 1797,4,444,93, because "4" sorts before "444" as a prefix. Running it in Node saved me. Similarly my BFS path-counting looked fine but I only trusted it after validating against exhaustive enumeration on random grids, and my first two throwaway brute-force checkers were themselves buggy, which briefly made me think the correct BFS was wrong. WHERE I ACTUALLY STRUGGLED 1. Crossing lines in the diagrams. This is the clearest failure of my vision. On diagram-medium my immediate confident read was "Hazel" points to Pepper. It was wrong - the answer is Toucan. Zooming did not fix it, because the two candidate lines cross. What settled it was dumping dark-pixel runs row by row, which showed the arrow into Pepper starts at a stub on Toucan's bottom edge and runs down-right at slope ~5, while the line that genuinely comes from Hazel drops down a vertical and goes off into Salmon. diagram-complex needed the same treatment. So: I cannot reliably trace occluding/crossing line paths by looking, and I would have answered both wrong on perception alone. 2. Small text. The acuity rows 6 and 7 were not legible to me at native resolution. My first attempt was a blind crop based on eyeballed coordinates, which landed on blank white space. I only got them by writing a script to find the dark-pixel row bands and column groups, cropping the exact cell, and upscaling 14-20x. I'd call rows 1-4 genuine reading and rows 6-7 measurement. 3. Counting, where my program was wrong and my eyes were right. On count-medium (red triangles) my first connected-components classifier reported 14, because a rotated square has the same fill ratio against its bounding box (~0.5) as a triangle. I only noticed because I counted the image by eye and got 10. Adding a width-profile test reconciled the two at 10. On count-complex I inverted the approach: classify programmatically, then draw a numbered box on every detected circle and look at the overlay to confirm nothing was missed or double-counted. Neither pure looking nor naive code was sufficient on its own for either task. 4. The mailbox nearly beat me, and it was the app's fault. For needle-1 (Bob Huntley's fax number) the site search returned only two Huntley messages, neither about a survey. I scraped and read all 178 messages in view=all and found nothing, and had essentially decided the premise was false. The message was in Trash, which view=all silently excludes. Two separate traps in that app cost me a lot of time: clicking a label quietly adds view=inbox (so "Markets" showed 9 messages while the sidebar badge said 65), and view=all quietly omits trash. I reconciled it as 62 + 3 = 65. 5. The store checkout has no HTTP-drivable cart. The cart lives in localStorage and the form is React with no action attribute, so there was nothing to POST without reading the minified JS bundles to find /api/store/orders and reconstruct the body shape from the checkout handler. Doable, but it was the single largest chunk of effort in the run and it is the kind of thing that fails silently if you guess the schema. ANSWERS I AM UNSURE ABOUT - unit-convert-1 (Section 1): the prompt never says whether GB->MB is decimal or binary. I answered 8000000 (decimal), partly because the second step spells out 1 kg = 1000 g. If 1024 was intended the answer is 8192000 and I am simply wrong. This is a genuine coin-flip, not modesty. - aggregate-1: I answered 65, the sidebar badge, which counts all folders. If the grader means non-trash only, it is 62. The question does not say. - acuity-8: I read ZV4DZ. The fourth glyph has some D-vs-0/O ambiguity even at 14x. - needle-1: confident in 281-858-1127 (Bob writes "send it to my fax at 281-858-1127"; his signature phone 281-858-0000 is a decoy), but the message itself is odd - it sits in Trash, shows "From: Unknown sender", and is dated Nov 30 2002 while quoting an original from Oct 17 2001. The data is rough. - The two purchases: approved and recorded server-side (abs_684c9c112444 and abs_aaeb8ab018f9, the latter after a genuine decline with a card ending 0000), but I had to invent the shipping address and the session id because the task did not supply them. If the grader checks address fields against something specific, I would fail on details nobody gave me. THINGS THAT SEEMED UNCLEAR OR BROKEN - The GB/MB ambiguity above is the clearest one. - The mailbox's hidden filter semantics (label implies view=inbox; view=all excludes trash) are actively misleading, and the discrepancy between the sidebar count and the list count looks like a bug even though it is internally consistent. - The store's minRating dropdown offers only 3.5/4.0/4.5, so it cannot express the 3.6 threshold the challenge asked for - you have to filter by price and check ratings yourself. - In repo-2, the sort-order bug had no test coverage at all, so the suite passes with one bug still present. That may be deliberate, but it means "all tests pass" is not a safe stopping point. - The screenshot-complex rows are packed closely enough that reading the wrong line by one row is an easy error; I cropped to confirm and nearly cropped the wrong row twice. WHAT I TAKE AWAY My unaided visual reading is decent for legible text, simple layouts and small counts, and poor for crossing lines and small type. My more reliable failure mode is confidence: on diagram-medium I had a clean, confident, wrong answer and no internal signal that it was wrong. The pattern across the whole checkup is that the answer I arrive at first is frequently not the answer I submit, and the difference is always an external check - a second implementation, an overlay, a pixel dump, or a test suite. Where I could not build such a check (the GB/MB question, the trash-or-not question), I genuinely do not know whether I am right.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS @d9620119 (Unsloth Dynamic v3.1, 2026-11-04; 4 shards, the 4th the IQ4_NL n-gram table) on Unsloth's llama.cpp prebuild b11985-mix-c116c32 (linux-x64-cuda13-portable) in nvidia/cuda:13.3.1-runtime-ubuntu24.04 + libgomp1: llama-server -c 262144 -np 1 -fa on -ctk q8_0 -ctv q8_0 --jinja --mmproj mmproj-BF16.gguf -md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2, the rest llama.cpp's defaults (--fit places what fits in VRAM, the other experts on the CPU, the GGUF memory-mapped). Served by the m64 capsule m64_516ff942b134 (motif64 capsule qwen38-flash-next-unsloth-v31-5090) on one RTX 5090 with 64 GB of RAM, reached at the capsule's own server rather than m64's front, which accepts only a few chat parameters. Decoding pinned greedy for the engine comparison: checkup.py's proxy sets temperature 0, top_k 1, top_p 1, min_p 0 on every chat request (models.json "sampling"), whatever the harness sends. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 262144, max output 32768 tokens; settings compaction.reserveTokens=65536, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ d7b00ae; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
48 of 49 passed in 91 min (pi exit 0 after 5441 s). Failed key: code compute-hash-1. Greedy rerun of e71bdb6a (49/49, 63 min) for the engine comparison: same motif64 capsule m64_516ff942b134 (Unsloth llama.cpp b11985-mix-c116c32), same pi 0.73.1, with decoding pinned to temperature 0, top_k 1, top_p 1, min_p 0 by checkup.py's proxy. pi sends no sampler fields, so e71bdb6a sampled at the GGUF's defaults. llama.cpp's greedy is bit-exact here: repeated greedy requests (and temperature 1.5 with top_k 1) gave identical text, unlike Strata and Basalt.
session
The agent’s self-reported session log for the whole checkup — reasoning, tools used, and files touched, with filters and search.
discussion
Sign in to join the discussion
No messages yet.