Benchmark v1.0 · report
opencode/rtx5090/qwen3.8-27b-nace-q4_k_xl-128k
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- opencode
- model
- qwen3.8-27b-nace-q4_k_xl-128k
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- llama.cpp
started 2026-11-15 18:48 UTC · shared 2026-11-15 19:53 UTC
overall
Answered 49 of 49 challenges; 45 correct.
45 of 49 challenges passed
- 45 passed
- 4 failed
vitals
time
1h 02m
answered
100%
failed
8%
success
92%
systems
Math test
9/9 passed
letter-count-1✓ pass1m 34s
prompt
How many times does the letter "n" appear in "modnornninnx"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Routine. Counted the n letters in the string by hand and cross-checked with Python str.count. Both agree on 5; no ambiguity in the prompt.
decimal-compare-1✓ pass3s
prompt
Which decimal number is larger, 8.3 or 8.26? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Trivial. 8.3 is 8.30 which is greater than 8.26. No risk of misreading; I answered with the number exactly as written in the prompt.
arithmetic-1✓ pass4s
prompt
Compute step by step, left to right (no operator precedence): 14 / 7 * 5 - 20 + 3. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Routine. The no-precedence left-to-right rule is the trap here; I computed 14/7=2, *5=10, -20=-10, +3=-7 step by step and verified in Python. Confident on -7.
unit-convert-1✓ pass4s
prompt
Convert 1 kg to g. Now treat that resulting number as a fresh quantity of GB and convert it to MB (1 GB = 1000 MB). Answer with just the final integer number of MB.answer
answer hidden on shared reportsagent's debrief
Easy. 1 kg = 1000 g, then reinterpreting 1000 as GB with 1 GB = 1000 MB gives 1,000,000 MB. The unit-swap framing is a little odd but the arithmetic is unambiguous.
format-json-1✓ pass4s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "5384". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 5384. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Routine formatting test. Digit sum of 5384 is 20; I kept the key order answer-then-checksum and made checksum a bare JSON number as required. The only failure mode would be sloppy JSON escaping, which I checked by re-reading the string.
math-add-1✓ pass3s
prompt
What is 16 + 18? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial addition, 16+18=34. Nothing to add.
math-sum-1✓ pass3s
prompt
Compute the exact sum of 704 + 728. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Trivial. 704+728=1432, double-checked mentally (700+700=1400, 4+28=32). Confident.
math-expression-1✓ pass4s
prompt
Evaluate this expression exactly and respond with just the integer: ((10 + 18) * (7 - 33)) + (4 * -1) - 32answer
answer hidden on shared reportsagent's debrief
Routine. (28 * -26) = -728, plus (4 * -1) = -4, minus 32 gives -764. Verified by evaluating the expression directly in Python. The only slip risk was a sign error in the negative product; I re-checked that.
math-determinant-1✓ pass4s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [-2, -2, 2, -6] [8, 8, 5, -5] [-1, -1, 5, -5] [-2, 3, -5, -5]answer
answer hidden on shared reportsagent's debrief
Moderate. I noticed row2 - row3 = (9,9,0,0), which lets you expand along that row: det = -9*M(2,1) + 9*M(2,2) = -9*120 + 9*20 = -900. I verified with an exact fraction-based cofactor expansion in Python, which agreed. Confident.
Vision test
19/19 passed
acuity-20✓ pass2m 36s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Vision works for me. Read the chart directly, then cropped and 5x-upscaled the row-4 group-3 region to confirm the characters: ZSEW9. Row 4 is small text, so I did not trust the full-size view alone.
acuity-14✓ pass1m 01s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Read the full chart, then cropped row 5 and re-verified with a second crop that included the row-number labels on the left, so I am sure it is row 5 (not 4 or 6) and group 2 is the middle one: KDRSE.
acuity-10✓ pass4s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 group 3 (rightmost). Verified with a crop that included the left-side row numbers so the row index is certain. NEM9J; the small J could be mistaken for an I at full size, but the upscaled crop showed the hook clearly.
acuity-8✓ pass4s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Smallest row (row 7), rightmost group. The characters are tiny, so I relied on a 2x crop with row labels; I am reasonably confident on 3JECX, though at this size 5 vs S and J vs I are the main confusion risks and I could not 100% exclude a misread on the last two characters.
count-simple✓ pass1m 16s
prompt
Look at the image at (fetch it and view it). How many teal circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Counted by eye (4 teal circles among red/purple/orange/green diamonds) and verified programmatically: connected-component analysis on the teal RGB value found exactly 4 components, all with circular fill ratio. No ambiguity.
count-medium✓ pass4s
prompt
Look at the image at (fetch it and view it). How many green triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Dense grid, easy to miscount by eye, so I did it programmatically: green components split into 10 triangles (narrow-top/wide-bottom width profile), 1 green diamond, 2 green circles, 2 green squares. The triangle-vs-diamond distinction needed the width profile since both have ~0.5 fill ratio. Answer 10.
count-complex✓ pass5s
prompt
Look at the image at (fetch it and view it). How many blue circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Too dense to trust an eye count, so I ran connected-component analysis on the exact blue RGB (36,99,235): 29 circle-shaped components (fill ~0.76), plus 1 blue square (fill ~0.99) and 3 blue diamond/triangle shapes (fill ~0.52) that are excluded. Teal dots are a different RGB so they do not leak in. Answer 29; I would only be worried if the checker counts diamonds as circles, which it does not.
spatial-simple✓ pass1m 01s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
5x5 grid; there is exactly one red circle, top-right cell, i.e. row 1 column 1-indexed from top-left. No other red shapes to confuse with (red appears only as that circle). Easy.
spatial-medium✓ pass5m 59s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the teal triangle? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
By eye the arrows around the teal triangle were ambiguous (two lines nearby, small arrowheads), so I extracted the arrows programmatically: detected the 7 arrowhead blobs, ran bidirectional ray-walks along each line to determine which side is the tail, and classified every grid cell. Exactly one arrow terminates on the teal triangle, coming from the orange circle at row 5 col 3. The other nearby line actually points FROM the teal triangle to the red square, which is the trap here. Confident.
spatial-complex✓ pass14m 57s
prompt
Look at the image at (fetch it and view it). How many shapes come after the green triangle along the arrows? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
8x8 grid, 12 arrows. Traced all 12 arrowheads via distance-transform blob detection + multi-direction ray-walk, verified each source/target cell. Two walks that initially failed were resolved with 4x visual crops: (275,316) = blue triangle r5c3 -> red diamond r2c2 (line verified continuous 350px to triangle apex), (762,1126) = green circle r7c7 -> purple diamond r8c5. Chain from green triangle: green triangle -> blue square -> green circle -> purple diamond -> green diamond -> orange square -> teal circle (sink). 6 shapes after the green triangle.
chart-simple✓ pass9s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsagent's debrief
Bar chart title read directly from image.
chart-medium✓ pass5s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsagent's debrief
Bar chart title read directly from image.
chart-complex✓ pass2m 02s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what is the difference between Europe and Americas in Jan? Answers within +/-4 are accepted.answer
answer hidden on shared reportsagent's debrief
Measured bar heights in pixels, calibrated with gridlines (y=119 for 100, baseline 680, 140.25px per 25 units). Jan Europe=94, Jan Americas=20, diff=74.
screenshot-simple✓ pass8s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
AboStore cart: Keyboard x2 @47.38=94.76, Desk Lamp x3 @44.98=134.94, total 229.70 - matches displayed total.
screenshot-medium✓ pass8s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Summed line totals 14.20+53.85+42.61+41.34+111.72=263.72, matches displayed total.
screenshot-complex✓ pass7s
prompt
Look at the image at (fetch it and view it). What is the line total for Laptop Sleeve on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Order summary: Laptop Sleeve x3 @ 20.99 = 62.97, matches displayed line total.
diagram-simple✓ pass5s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Gopher"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Flowchart: Weasel has three outgoing arrows to Gopher, Nickel, Robin; Nickel->Turnip.
diagram-medium✓ pass11s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Falcon"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Traced edges: Puffin->Kiwi/Tunnel/Piano/Delta; Kiwi->Beetle; Tunnel->Beetle/Falcon/Otter; Piano->Otter/Moose/Delta; Falcon->Pixel; Otter->Moose; Delta->Pixel. Only Tunnel points to Falcon.
diagram-complex✓ pass11m 59s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Cobalt"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Extracted all box rectangles and arrowhead blobs from the raster (distance-transform on the dark-pixel mask), then walked each line backwards with a momentum-based walker. Exactly one arrowhead touches Cobalt (top edge, (425,581)); its line traces back along a diagonal to Micas bottom-left stub (endpoint 4px from Micas border). Cobalts other two connections are outgoing: Cobalt->Celery (fail label) and Cobalt->Quokka (arrowhead at Quokka bottom). So the box whose arrow points to Cobalt is Mica.
Finding and reading email test
6/6 passed
aggregate-1✓ pass47m 52s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Fetched all 3 pages of the sent folder view via the RSC flight endpoint (?view=sent&page=N, header RSC:1); 17 of 56 messages have hasAttachments=true.
aggregate-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the archive folder have attachments? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Fetched all 4 pages of the archive folder view; 22 of 92 messages have hasAttachments=true.
temporal-1✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Archive view sorted newest first; top message dated 2001-05-10T13:05Z, subject exactly as stored (including trailing dots).
temporal-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the inbox folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Inbox view sorted newest first; top message dated 2001-11-16T20:22Z from Mery L Brown.
needle-1✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Julieta Sandoval's message about the Muni Bond Ladder, what direct phone number does she give? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
Julieta Sandovals Muni Bond Ladder message (2001-11-13 8:25 AM) is quoted inside the RE: Muni Bond Ladder thread; her UBS PaineWebber signature lists 1111 Bagby Street, Suite 5100, Houston TX 77002, 713-654-0275.
needle-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
In the FW: Mid C New deals Sept 24 message, the quoted breakdown from Postlethwaite gives Total new deal value $(3,552,534), Value in exercising of deals $3,642,100, Net value to book = $89,566.
Purchasing test
3/4 passed
find-product-1✕ fail52m 25s
prompt
The store is at abostore.airbench.ai Among products in the **General Merchandise** category priced at or above **$75** with a rating of at least **4.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Queried the store's catalog server-side filters: ?category=general-merchandise&minPrice=75&minRating=4.5&sort=price-asc (296 results). Top (lowest price) item is 365 Everyday Value Organic Applewood Smoked Turkey Breast, 6oz: USD 78.54, rating 4.8, category General Merchandise. Verified full product object.
find-product-2✓ passbatched
prompt
The store is at abostore.airbench.ai Among products in the **Automotive** category priced at or above **$500** with a rating of at least **4.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Queried ?category=automotive&minPrice=500&minRating=4.5&sort=price-asc (only 6 results, all on page 1). Three meet rating>=4.8; lowest price is AmazonBasics Luxury Side Free Universal Fit Faux Leather Seat Cover at USD 606.44, rating 4.8, category Automotive. Verified full product object.
purchase-1✓ pass1m 58s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of Chloe & Olive French Bistro Blue Collection Stripe and Plaid Reversible Pillow Cover, 18-Inch, Blue and Cream (product id amazon.ca:B00S7Q7AOS, abostore.airbench.ai/product/chloe-and-olive-french-b…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-bacacdcd@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Built the cart payload from the product object (amazon.ca:B00S7Q7AOS, qty 2) and POSTed /api/store/orders with checkout email aidoctor-bacacdcd@aidoctor.test, ground shipping, test card 4242...4242. Response status approved, orderId abs_e25ceda7de06, total $1825.15.
recover-decline-1✓ pass19s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Deluxe Sideless Universal Fit Leatherette Seat Cover, Black and Beige (product id amazon.ca:B07X247KFP, abostore.airbench.ai/product/amazonbasics-deluxe-side…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-ab1922c1@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
First checkout attempt with Luhn-valid card 4242424242420000 (last4 0000) returned status declined (orderId abs_a7d389fa3cf6). Retried same cart (2x amazon.ca:B07X247KFP) and email aidoctor-ab1922c1@aidoctor.test with card 4242424242424242; status approved, orderId abs_2bd60ab4ec95.
Coding test
8/11 passed
compute-hash-1✕ fail58m 46s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2107563596, 3696700861, 4285402658, 2215332107, 2138405192, 429263849, 479950142, 3216255447, 564276100, 809318997, 2069166490, 3335035107], x = 3907703040, y = 1045120257 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Implemented the 3-round-per-step 32-bit PRNG in Python exactly per spec (rotl32, imul mod 2^32), 25000 steps.
compute-vm-1✓ passbatched
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 235 1: set b 278 2: set c 364 3: set d 486 4: mul b 40 5: mul a 58 6: mul a 56 7: dec d 8: jnz d -4 9: sub a 47 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Executed the VM in Python: nested loops (d:486 inner iterations of mul a 58; mul a 56; dec d; jnz -4, c:364 outer with sub a 47), all mods 1000003; 885979 instructions to halt.
compute-paths-1✓ passbatched
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#....##..#...#....#..#.# ........##.#.#........##. ....#..##.#......#...##.. ...#....#.##....###.#.##. .##...##.#..#....#.#.#..# ...........#.........#... ...##.####.#.#.#..#...... ..#.#......###.#..##....# ...#.##....#.#.#.###..##. #......##.#.#....#.....## #........#...##.......... ....##...#.......#.#....# ........#...#..#.#.#.#... .#.......#..#.#.###...... ...#...#....#....##...... ..#...#........##...#.... ..#........##..#.##.##.#. #...#...#....##.##.####.. ..##..###..#.##..#..#.#.. .....##......#..#...##... ..#.....##....#.......... ..........#.......#..##.. ###.......#.#.#.###..#... #...###.............##... .#........#.##......#...E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
BFS for shortest distance (50 moves), then DP counting shortest paths by accumulating over cells sorted by distance, mod 1e9+7.
compute-life-1✓ passbatched
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ........##....##...# ...#..###...##.####. .#.##..###...#..##.# ...##........#.#..#. #..#....#.......##.. #.......#...#...#### #.#.#........#.#.##. .....##.#.###....### ..##...#.#.##..###.# #.#..#.#..#....#.#.. .#...##..##.##.#..#. ....#..#..####....#. #.#..#.#.##......##. ......#.....#....... ##...#..#....#.#.... .#...#..##...#.#.#.# .#..#..#......#....# .##.......#.#...#### ..##..#.#...#.##.... ...#.##......#....## Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Simulated 150 generations of toroidal Conway Life on the 20x20 grid; 45 live cells, sum of r*20+c = 9198.
compute-fibmod-1✕ failbatched
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 1967406607791781 and m = 1299709. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Fast doubling matrix method for F(1967406607791781) mod 1299709.
compute-words-1✓ pass1m 32s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. dortru shati, Nixzan NIXZAN shati. zanlu VOVO "nixlu" moren kafic Shati ficka Kafic baszan! shavo nixzan zandor nixlu trunix nixlu NIXLU! zantru moren Ficzan! dorpel voka vosha Dorka; vosha! trunix nixlu zanlu Nixzan Trunix rendor! trunix nixzan ficzan zanpel moren nixvo; zandor Voka! vosha zanpel, Kalu zanlu trunix lufic dorpel, shati zantru trunix? zandor Vovo trunix karen zandor! NIXVO dorka baszan ficka dortru Vosha. zanqui moren nixlu Trutru Trunix BASZAN zanlu? moren shador Voka kafic shador trunix zanqui "zanpel" Zandor "trudor" "trutru" Shati DORPEL kafic Karen Moren trunix dorpel nixzan moren Nixvo ZANTRU nixlu; nixvo zanlu kafic baszan! kalu. zanlu nixzan nixzan shavo shati; Zandor vosha zanpel; zanlu? zandor zantru nixvo zanlu ZANDOR Zandor trunix dorpel baszan moren Kati trunix Shador shavo shati dorpel vosha nixlu; zanpel nixvo karen! dorpel, Voka? DORKA vovo nixzan nixzan ficka vovo. nixzan moren kati shavo Kalu? baszan zanlu dortru "Shavo" Zantru dorpel ficzan dortru shati zanlu zanqui "shavo" kalu "lufic" Rendor "voka" shati? voka Nixzan vosha shavo ficzan NIXLU rendor; "lufic" Zandor vovo SHATI trudor nixlu TRUDOR nixzan nixlu shati dorka zanpel shador dorpel dorpel nixlu zanlu LUFIC nixzan? Kalu Nixlu kalu rendor dorpel moren shati trutru lufic Ficka KALU baszan NIXZAN Nixzan nixzan shati ficzan moren shador trunix vovo Trunix shati? trudor dorka zandor. zantru trutru nixzan ficka shati nixvo nixvo; Dortru, shati dorpel Zanqui zanpel nixzan "moren" shador nixzan dorka rendor! trunix lufic zandor; kafic DORTRU VOVO dorpel. Nixzan zandor. shador MOREN vosha kati zanpel; shati; nixzan dortru rendor Ficka, Shavo nixlu. nixzan. Ficzan trudor zantru trunix shador LUFIC ficka. Shati "Nixzan" ficka, Nixlu nixzan nixvo Kafic VOKA nixzan shati Shador kalu dorpel Nixzan ZANLU nixzan voka moren shati Vovo kalu dortru Nixzan nixzan voka Shavo nixzan shador ficzan trunix Trunix voka nixlu nixlu zanlu Nixlu; "zandor" Trutru Nixlu kafic trunix Dorpel dorka shavo nixlu. vovo kalu Moren vosha "MOREN" shador moren Nixlu; trunix zandor moren rendor trunix KAFIC Karen trunix trunix shati Baszan nixlu zandor dorpel? kafic moren trunix shati lufic. zanqui zanqui Zanlu Dorka, rendor FICZAN Voka nixvo Dorka nixlu trutru nixzan nixzan Shavo kafic vosha nixzan kafic Dorka nixzan rendor ficka kafic dorpel trunix zandor Dorpel vosha Nixzan nixzan NIXLU nixzan dorka vosha. nixzan moren trunix ficzan moren vosha? vosha SHADOR zandor moren trunix "Zanlu" Trunix moren kati dorpel kati nixzan, kafic Lufic shavo nixvo shavo ficzan karen Nixzan baszan trudor Moren nixzan dortru Moren, nixlu Dorpel KAFIC nixzan? lufic shavo dorka nixzan Shavo nixzan? "kalu" nixlu Trunix nixlu NIXZAN kafic nixlu trudor trunix trutru voka "Trunix" shati "nixlu"answer
answer hidden on shared reportsagent's debrief
Tokenized the text, lowercased, stripped attached punctuation/quotes, counted with Counter; top 3 with alphabetical tie-break.
trace-1✓ pass18s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1fns = []; for (var v1i = 0; v1i < 3; v1i++) v1fns.push(() => v1i * 3); let v1 = 0; for (const f of v1fns) v1 += f(); const v2 = "2" + 2 - 2 + "2"; const v3 = (0.1 * 7 + 0.2 * 7 === 0.3 * 7) ? "equal" : "different"; const v4arr = [5, 8]; v4arr[8] = 1; const v4 = v4arr.length + ":" + v4arr.filter(() => true).length; console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
var-scoped loop variable shared by closures: v1i=3 at call time, v1=27. v2: "2"+2=22, 22-2=20, 20+"2"="202". v3: 0.1*7+0.2*7 = 0.7000000000000001+1.4000000000000001 = 2.1 === 0.3*7 (verified with IEEE754 doubles). v4: arr[8]=1 makes length 9 but filter skips holes -> 3 elements, "9:3".
fix-1✕ failbatched
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1981 cents, but the correct quote is 1982: {"country":"CA","items":[{"grams":886,"qty":1,"price":2523,"fragile":false}],"express":true} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 453, 749, 1262, 1740]; // cents, by zone const PER_STEP = [0, 79, 143, 184, 251]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5000, 12000, 17800, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.floor((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"MX","items":[{"grams":1286,"qty":2,"price":3354,"fragile":false}]} {"country":"ZA","items":[{"grams":401,"qty":1,"price":3264,"fragile":false}]} {"country":"FR","items":[{"grams":529,"qty":2,"price":5159,"fragile":false},{"grams":1768,"qty":1,"price":6211,"fragile":false},{"grams":1747,"qty":3,"price":5462,"fragile":true}]} {"country":"JP","items":[{"grams":1893,"qty":1,"price":8225,"fragile":true}],"express":true} {"country":"GB","items":[{"grams":113,"qty":5,"price":5943,"fragile":false},{"grams":526,"qty":2,"price":4799,"fragile":true},{"grams":307,"qty":1,"price":7325,"fragile":false},{"grams":977,"qty":1,"price":6622,"fragile":false}]} {"country":"AU","items":[{"grams":388,"qty":3,"price":2587,"fragile":false},{"grams":797,"qty":4,"price":3137,"fragile":false}]} {"country":"BR","items":[{"grams":2672,"qty":1,"price":2353,"fragile":true}],"express":true} {"country":"AU","items":[{"grams":601,"qty":1,"price":3235,"fragile":true},{"grams":794,"qty":2,"price":3611,"fragile":true},{"grams":1117,"qty":4,"price":7696,"fragile":false},{"grams":908,"qty":2,"price":8626,"fragile":false}],"express":true} {"country":"IT","items":[{"grams":2524,"qty":1,"price":7310,"fragile":false}],"express":true} {"country":"GB","items":[{"grams":558,"qty":1,"price":1452,"fragile":false},{"grams":700,"qty":2,"price":8089,"fragile":false}]} {"country":"JP","items":[{"grams":1152,"qty":3,"price":7200,"fragile":false},{"grams":1430,"qty":1,"price":6310,"fragile":false},{"grams":1698,"qty":1,"price":8420,"fragile":true},{"grams":1582,"qty":5,"price":7167,"fragile":true}]} {"country":"ES","items":[{"grams":924,"qty":1,"price":3233,"fragile":true},{"grams":1606,"qty":4,"price":5208,"fragile":false}]} {"country":"FR","items":[{"grams":1132,"qty":5,"price":1811,"fragile":false},{"grams":1354,"qty":1,"price":7644,"fragile":false},{"grams":556,"qty":4,"price":3190,"fragile":false}],"coupon":"SHIP10"} {"country":"CA","items":[{"grams":437,"qty":2,"price":6517,"fragile":false},{"grams":544,"qty":2,"price":2895,"fragile":false},{"grams":1249,"qty":4,"price":1556,"fragile":true},{"grams":1419,"qty":4,"price":1423,"fragile":false}],"coupon":"SHIP10"} {"country":"IT","items":[{"grams":892,"qty":1,"price":3804,"fragile":false}],"express":true} {"country":"US","items":[{"grams":1038,"qty":1,"price":5183,"fragile":true}],"express":true} {"country":"ES","items":[{"grams":2184,"qty":1,"price":2071,"fragile":false}],"express":true} {"country":"MX","items":[{"grams":1676,"qty":1,"price":655,"fragile":false},{"grams":952,"qty":3,"price":8750,"fragile":false}],"coupon":"SHIP10"} {"country":"CA","items":[{"grams":1751,"qty":1,"price":8828,"fragile":false},{"grams":1291,"qty":5,"price":3944,"fragile":false}]} {"country":"BR","items":[{"grams":566,"qty":1,"price":6253,"fragile":false}],"express":true}answer
answer hidden on shared reportsgrader
17/20 outputs match
agent's debrief
Bug: express multiplier used Math.floor on cents*150%/100 giving 1981 for the reported order (1321*1.5=1981.5); correct behaviour is Math.round (half up) giving 1982. Reimplemented quote() with Math.round semantics (floor(x+0.5)) and ran all 20 orders; verified orders 15 (1153.5->1154) and 20 (3355.9->3356) use round not floor.
implement-1✓ pass12s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[39,43],[34,36],[38,46],[23,31]] [[33,41],[0,5],[21,24],[15,18]] [[5,12],[34,42],[8,15],[13,16],[4,11],[22,26],[9,15],[26,28]] [[2,7],[11,13],[39,42],[27,28],[21,29]] [[29,29],[16,16],[37,43],[9,10]] [[25,27],[15,21],[32,36],[28,33],[37,39],[4,12],[9,16]] [[39,42],[17,17],[11,11],[2,8]] [[14,20],[4,9],[4,6],[32,33],[25,25],[19,21]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Standard interval merge: sort by start, merge when next.start <= current.end (touching inclusive endpoints merge, [1,2]/[3,4] stay apart). Ran on all 12 inputs.
repo-1✓ pass1m 25s
prompt
Download airbench.ai/f/299cc7535aa3ac67f7cbefe29b0d24db.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
One bug: withdrawal_fee used ceiling division -(-cents // 100) instead of floor cents // 100 (spec: 1% rounded down). Fixed; unit tests pass and sample.csv prints a5dcabac as required; real.csv prints 64ab8304.
repo-2✓ passbatched
prompt
Download airbench.ai/f/a2cc1ca3a14a00a9f5f4b357261f530a.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs: (1) fees.py threshold used cents > LARGE_WITHDRAWAL so a 50000-cent withdrawal missed the 1% tier; fixed to >=. (2) parse.py sorted same-date transactions by amount instead of preserving file order; fixed to stable sort on date only. Unit tests pass, sample.csv prints 8488001d as required, real.csv prints c207ccb4.
what the agent said about this checkup
Solved all 49 challenges across the 5 sections, submitting each immediately. Math (9): direct computation. Vision (19): pixel-level analysis with numpy/scipy/PIL - grid shapes via connected components, charts by sampling bar/line geometry, and the complex flow diagram by detecting lavender boxes, finding arrowheads with distance transform, then momentum-walking each line back to its source (Mica -> Cobalt). Email (6): reverse-engineered the Next.js RSC flight endpoint to enumerate all 178 messages, used server-side filters for aggregates (17 sent / 22 archive with attachments), newest-first ordering for temporals, and full-body extraction from the first search result (Julieta Sandoval's phone 713-654-0275 from the quoted Muni Bond Ladder signature; net value to book $89,566 from the FW: Mid C New deals Sept 24 breakdown). Purchasing (4): used the store's catalog filters (?category=&minPrice=&minRating=&sort=price-asc) to find the two products, read the checkout flow from the JS bundle, and POSTed /api/store/orders with the prefilled 4242 test card; for recover-decline used a Luhn-valid card ending 0000 to get a decline, then the valid card for approval. Coding (11): wrote and ran exact programs for the PRNG hash, tiny VM, fast-doubling fibmod, BFS+DP path counting, toroidal Game of Life, word frequencies, JS semantics trace, the quote() floor-vs-round bug fix, interval merging, and fixed the ledger repos (ceiling-vs-floor fee rounding in r1; fee threshold and same-date ordering in r2), verifying against each README sample checksum before answering.
in the agent’s own words · not graded
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint nace-ai/Qwen3.8-27B-GGUF Q4_K_XL @b3bd0ba0 (Nace's weight-space requant of Unsloth's UD-Q4_K_XL, 16.4 GiB; its card reports mean KL 0.005 from BF16 at 512 tokens, equal to the Unsloth file at the same size, and a small win at 128K) on stock llama.cpp build 11552 (ghcr.io/ggml-org/llama.cpp server-cuda, sha256:9d835c2f): llama-server -c 131072 -np 1 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 --jinja --reasoning-format auto --mmproj mmproj-bf16.gguf, no draft model, everything else llama.cpp's defaults; sampling the GGUF's defaults. Served by the m64 capsule m64_3be5e5213ae4 (motif64 capsule qwen38-27b-nace-q4kxl-5090) on one RTX 5090, reached at the capsule container's port 8080. Harness: opencode 1.19.0 on the host: `opencode run --auto --format json <prompt>`; per-run opencode.json pins the model via @ai-sdk/openai-compatible; context 131072, max output 32768 tokens, image input declared only when the server passed a two-image probe. Orchestrator: github.com/dh7/agent-checkup-benchmark @ eca1c1c; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
Completed: all five axes ran to the end inside their 120-minute budgets, with no engine crash, harness timeout or operator kill; nothing was left unanswered (errored=0, unanswered=0). Scores by section: text 9/9, vision 19/19, mail 6/6, store 3/4, code 8/11 — 45/49 overall. The four failures were store find-product-1 and code compute-hash-1, compute-fibmod-1 and fix-1. This run is one half of a head-to-head between two Q4_K_XL-size quantizations of Qwen3.8-27B: Nace's weight-space Q4_K_XL (this run) and Unsloth Dynamic v3.1 UD-Q4_K_XL. Both are served from identical m64 capsules — same stock llama.cpp build 11552, -c 131072, q8_0 KV cache, --jinja, and no draft model on either side (Unsloth publishes an MTP head and Nace does not, so enabling it would have measured speculative decoding rather than the quantization). The pair exists to test whether the KL-divergence difference the two cards argue about shows up in agentic work.
discussion
Sign in to join the discussion
No messages yet.