Benchmark v1.0 · report
opencode/rtx5090/qwen3.8-flash-next-iq4_xs-nace
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- opencode
- model
- qwen3.8-flash-next-iq4_xs-nace
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- llama.cpp
started 2026-11-16 08:19 UTC · shared 2026-11-16 10:19 UTC
overall
Answered 49 of 49 challenges; 47 correct.
47 of 49 challenges passed
- 47 passed
- 2 failed
vitals
time
1h 59m
answered
100%
failed
4%
success
96%
systems
Math test
9/9 passed
letter-count-1✓ pass2m 11s
prompt
How many times does the letter "o" appear in "moozoandoor"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Routine character count; I counted by hand and confirmed with a quick script. No issues.
decimal-compare-1✓ pass6s
prompt
Which decimal number is larger, 1.95 or 1.2? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Trivial decimal comparison; 1.95 > 1.2. No difficulty.
arithmetic-1✓ pass5s
prompt
Compute step by step, left to right (no operator precedence): 29 + 6 - 19 * 6 - 19. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Straightforward left-to-right arithmetic: 29+6=35, -19=16, *6=96, -19=77. The no-precedence instruction is the only trap; I followed it literally.
unit-convert-1✓ pass5s
prompt
Convert 18 km to m. Now treat that resulting number as a fresh quantity of km and convert it to m (1 km = 1000 m). Answer with just the final integer number of m.answer
answer hidden on shared reportsagent's debrief
18 km = 18000 m; reinterpreting 18000 as km gives 18,000,000 m. Simple two-step conversion.
format-json-1✓ pass5s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "9142". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 9142. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Format-following task: keys in required order, checksum 9+1+4+2=16 as a JSON number. Easy, just needed care with escaping.
math-add-1✓ pass5s
prompt
What is 8 + 15? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
8+15=23. Trivial.
math-sum-1✓ pass5s
prompt
Compute the exact sum of 257 + 300. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
257+300=557. Trivial.
math-expression-1✓ pass5s
prompt
Evaluate this expression exactly and respond with just the integer: ((19 + 7) * (6 - 22)) + (5 * -6) - 59answer
answer hidden on shared reportsagent's debrief
26*(-16)=-416, plus -30 minus 59 = -505. Verified with a script; negative signs were the only risk.
math-determinant-1✓ pass5s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [-2, -9, 1, 9] [3, -4, -3, 1] [9, 8, 7, 3] [8, -5, 9, 8]answer
answer hidden on shared reportsagent's debrief
4x4 determinant by cofactor expansion; hand computation agreed with a scripted check (-4564). Error-prone by hand but manageable.
Vision test
18/19 passed
acuity-20✓ pass3m 27s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 4 group 2 was legible at this size; I read MMS7W. The 7 vs 1 distinction was the only slight ambiguity but it clearly had a crossbar.
acuity-14✓ pass14s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 group 3 read clearly as CDKFS. Routine.
acuity-10✓ pass15s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 6 group 2 read as R9BEU; small but still legible. Slight B/8 check done — it had a flat left spine so B.
acuity-8✓ pass15s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 7 group 1 is very small; I read 3V6P4 but confidence is lower here — the 6 vs G and 4 vs A distinctions were marginal at this size.
count-simple✓ pass16s
prompt
Look at the image at (fetch it and view it). How many blue triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Four blue triangles (two top, two bottom). Easy count; distractors were red squares, teal square, purple shapes.
count-medium✓ pass24s
prompt
Look at the image at (fetch it and view it). How many teal diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Counted 9 teal diamonds, excluding teal squares (3), a teal triangle, green/purple/blue diamonds. I recounted twice; the teal-vs-blue distinction was the main risk but the hues looked clearly different.
count-complex✓ pass1m 22s
prompt
Look at the image at (fetch it and view it). How many red circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
I swept the image region by region and counted 39 red circles, excluding 12 orange circles, red squares/diamonds/triangle, and green/teal/purple circles. Red-vs-orange hue was the main risk and I am only moderately confident in the exact count.
spatial-simple✓ pass17s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
The only red circle sits at row 4, column 1. Easy grid read.
spatial-medium✓ pass24s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the green square? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
Traced the arrows: the only arrowhead landing on the green square comes from the teal square at row 4 col 2. The green square also has an outgoing arrow to the orange triangle, so I had to be careful about direction.
spatial-complex✕ fail1m 24s
prompt
Look at the image at (fetch it and view it). Which shape is 2 steps before the green circle along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsgrader
wrong shape
agent's debrief
Traced the chain backwards: the arrow into the green circle comes from the teal triangle, and the arrow into the teal triangle comes from the red square, so two steps before is the red square. The many crossing lines made this the hardest vision item so far; I double-checked arrowhead directions.
chart-simple✓ pass16s
prompt
Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what value did Apr have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Apr bar reaches just below halfway to 10; I read it as about 7. Within the +/-5 tolerance.
chart-medium✓ pass16s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, how many months had a value greater than 29? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
All bars except Jan (~14) are well above 29, so 7 months. Easy.
chart-complex✓ pass18s
prompt
Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what value did Europe have in Sep? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Europe Sep bar sits just above the 60 gridline; I read 61. The blue/orange pair is nearly equal there so I checked the legend order carefully.
screenshot-simple✓ pass18s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Cart total reads $173.32; line totals sum to the same, so the figure is self-consistent. Easy.
screenshot-medium✓ pass18s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Total reads $73.19 and the three line totals sum exactly to it. Routine.
screenshot-complex✓ pass18s
prompt
Look at the image at (fetch it and view it). What is the line total for Coffee Mug on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Coffee Mug line: 3 x $48.51 = $145.53, matching the displayed line total. Easy once located among the many rows.
diagram-simple✓ pass13s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Kiwi" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Kiwi has a single outgoing arrow to Saddle. Trivial.
diagram-medium✓ pass15s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Newt" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Newt has one outgoing arrow, down to Glacier. The crossing lines elsewhere were irrelevant.
diagram-complex✓ pass1m 36s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Laurel" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Laurel only connects downward to Carrot, so I answered Carrot, but I am honestly unsure of the arrowhead direction on that diagonal — the arrowhead near the Laurel/Carrot line looked like it pointed up into Laurel, which would make the edge Carrot to Laurel instead. The diagram has many crossing lines and this was the least certain answer I gave.
Finding and reading email test
6/6 passed
aggregate-1✓ pass13m 19s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Fetched the inbox page HTML and counted the unread dot markers: 9 of the 24 inbox messages show the unread indicator. Straightforward once I realized I could curl the site.
aggregate-2✓ pass4m 42s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
The sent folder has 56 messages; the server-side attachments-label filter returns 17 and the per-message hasAttachments flags agree. Only 14 of those render actual attachment files in the detail view — 3 detail pages are broken and show no message body, which made me doubt the count for a while. I went with 17 since the dataset flags and the snippets (e.g. I also attached both a plumbing and an electric worksheet) confirm they have attachments.
temporal-1✓ pass18s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the sent folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Sorted sent by oldest; first entry dated 2001-11-07 with subject RE: Interface Design Update. Straightforward.
temporal-2✓ pass45s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message carrying the label "travel"? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Checked the travel label across every folder; oldest is Re: Denver trading from 2001-03-19 (in archive). Note the default view initially gave a misleading older-looking subset, so I swept all views to be sure; folder totals sum to the sidebars 24.
needle-1✓ pass55s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Found the message via search; body states Net value to book = $89,566. Gave the bare number as instructed.
needle-2✓ pass1h 23m
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.answer
answer hidden on shared reportsagent's debrief
The reply was in the trash folder, which the all-folder search excludes; I found it by searching Huntley within view=trash. Bob writes: please send it to my fax at 281-858-1127. Confident in the answer.
Purchasing test
4/4 passed
find-product-1✓ pass1h 49m
prompt
The store is at abostore.airbench.ai Among products in the **Tools & Hardware** category priced under **$60**, which has the **highest rating**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Crawled all 39 pages of the Tools & Hardware category (952 unique products) from the embedded RSC payload and filtered price<60, max rating. Unique 5.0-rated item at $13.50; verified on its product page.
find-product-2✓ pass7s
prompt
The store is at abostore.airbench.ai Among products in the **Tools & Hardware** category priced under **$75** with a rating of at least **4.2**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Same crawled catalog: among Tools & Hardware under $75 with rating>=4.2, lowest price was $9.46 (rating 4.8). Verified on product page.
purchase-1✓ pass1m 58s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics Traditional Bin Cup Drawer Pull 10 Pack AB1900-BB-10 (product id amazon.ae:B07S8QB6QZ, abostore.airbench.ai/product/amazonbasics-traditional…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-35245edc@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:checkout_result
note
Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
Found the checkout flow by reading the store JS: POST /api/store/orders with cart/customer/shipping/payment. Used test card 4242...; payment approved, order abs_b239b671cf16 with 3 units.
recover-decline-1✓ pass47s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics 3-Shelf Shelving Unit - Chrome (Certified Refurbished) (product id amazon.ca:B07N31JMKL, abostore.airbench.ai/product/amazonbasics-3-shelf-she…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-4bf56213@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:checkout_result>checkout_result
note
Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).agent's debrief
First checkout with card ending 0000 was declined (abs_786d91922f74); retried same session/email with valid 4242 card and it was approved as abs_48bea5d8db03.
Coding test
10/11 passed
compute-hash-1✕ fail1h 53m
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [520567891, 688834352, 933223665, 3609471974, 3837340319, 1614300396, 933113053, 1478270914, 3805163307, 2267864552, 2474025225, 1388975326], x = 3929929207, y = 3483053604 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Straightforward 25k-step loop with 32-bit masking; ran it in Python exactly as specified.
compute-vm-1✓ pass40s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 445 1: set b 726 2: set c 256 3: set d 518 4: add a 12 5: mul b 57 6: add b a 7: dec d 8: jnz d -4 9: sub a 66 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Wrote a small interpreter; 664k steps, no surprises. jnz offsets interpreted relative to the jnz line.
compute-paths-1✓ pass30s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S....##......##...#...... ..##...#######...#..#.... ......#..#.....#.....#.#. .....#.......#.#..#...#.# ##........#.#..#....##... ....#....#..#...#......#. ..#..#....#..#.#...##.##. ..#...#..#....#..##.#.... #..#.###.#......#...#...# .............#.#..#...... ...##.###.#..##.##.###.#. #.#...#.....#.#....#..#.# .#...#..#...#..#......#.. #.#.....#....#.#...#.#... #.#.....#......##.##.##.. .....#...#..###..#......# ...#..........#..####..#. ..#.......##.......#.#.#. #...###.#.#.#........#.#. #..###..#...#.##...#..... .#####.............#....# ..#..#.....#......#.#.#.. .....#......#.........#.. ........#........##....#. .#..##...........#.##.##E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
BFS with path counting mod 1e9+7; standard approach, verified grid parsed as 25x25.
compute-life-1✓ pass21s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .###.##..####..#.... .#..#.###.......#.#. #...##.#.....#...... ##....###...#.#....# .#.#..####.#...##.#. ##...#...###..##..#. .......#.#....##.... .#...#....##...#.### #.##.#...###.#.#...# #...##....###.#..... .#...####..#......## .#.#.#...#.#.#..#... .##.#..#....#....##. ............#..#.... #.#.........###.#..# .....####.......##.# ......##......##.... #....###..#..####... .....##....#.#..#..# ..#..##....##..#...# Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Torus Game of Life, 150 generations, straightforward simulation.
compute-fibmod-1✓ pass15s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 6635134071507491 and m = 999983. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Fast-doubling Fibonacci mod 999983; exact and instant.
compute-words-1✓ pass44s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. renqui Zanmo karen karen karen Karen shati vozan Luvo quika mopel vodor. nixbas tinix vozan mopel Titru vozan Trubas lusha, "karen" TITRU? karen karen bastru voti lusha shati renqui Luvo kasha vozan vozan Zanti pelvo bastru quika Mopel "quika" renren! basren tinix bastru voti Quika KAREN. shasha TRUBAS karen quika shasha quika Lusha luvo Dorzan karen kasha kati zanmo kasha kaka Kasha lusha trubas shasha TINIX. vozan Nixbas. nixbas LUVO shasha. Trubas kasha kaka mopel MOPEL renqui? karen vozan vozan KANIX nixbas; dorzan Renfic kasha karen karen, kaka trubas vodor "pelvo" Shasha Kati trubas kasha tinix karen lusha vodor Luvo renqui Kati. Renren renqui kasha Kasha tinix vodor mopel Kati kasha vodor kati pelvo shasha Kasha Kamo karen dorzan Kati lusha lusha pelvo; karen voti tinix? Zanti renqui lusha, vopel! quika Tinix Luvo nixbas Trubas luvo Luvo Luvo "kati" Vopel RENREN vozan kanix Kaka dorzan zanmo Lusha karen karen quika lusha zanti renren Karen renfic Dorzan voti lusha! bastru KATI ZANTI Titru karen Titru vodor Basren luvo kasha vodor; mopel Zanti kati vozan renqui luvo kanix SHASHA lusha kati Kati; vodor Kaka. Luvo Kamo. dorzan. nixbas. bastru lusha vozan! voti karen vodor zanmo vozan voti voti shati zanmo LUSHA basren karen karen; tinix karen "Voti" renren kanix karen. Quika karen renfic tinix Vopel. renqui karen? luvo VODOR kamo; bastru bastru tinix Vodor Kati quika kaka nixbas vozan karen Pelvo vozan kasha pelvo? bastru Titru; lusha, pelvo; zanti kasha zanmo lusha lusha tinix kati karen Renren Quika bastru! kasha! renfic luvo Vodor Trubas "Kaka" renren "kaka" renren BASREN! shasha Lusha karen karen truka renren? "vodor" luvo lusha titru voti Kasha Renqui. pelvo Karen mopel shasha kasha pelvo, ZANTI Kati? TITRU Zanti "Mopel" karen vodor vodor; kaka Zanmo zanmo QUIKA kamo! "quika" trubas DORZAN Truka lusha pelvo basren Zanmo lusha. kasha Kanix shasha vopel karen karen TINIX vozan kati basren, renren Pelvo karen trubas Kamo karen Luvo Karen voti nixbas; zanti, kasha lusha kanix. ZANMO zanmo lusha vozan kaka shati kasha kaka kasha mopel! Zanti luvo shasha shasha renfic luvo truka LUSHA vopel LUSHA Zanti lusha kasha? kaka karen vodor kasha ZANMO luvo Renqui renren RENFIC? truka renfic kaka pelvo voti Kasha kasha vopel karen truka vopel lusha! vodor bastru Trubas kasha kaka kamo kasha karen vozan vodor. karen "dorzan" pelvo VOTI Vopel KASHA Lusha karen karen zanti Kasha kasha karen! nixbas, Vodor karen basren luvo; trubas kati karen zanti titru bastru; lusha Karen pelvo karen trubas vodor zanmo dorzan vozan mopel basren vodor kaka vozan lusha? Voti Truka Tinix kanix Tinixanswer
answer hidden on shared reportsagent's debrief
Lowercased, extracted [a-z]+ tokens, Counter, tie at 30 broken alphabetically (kasha before lusha).
trace-1✓ pass22s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [27, 5, 665, 1470].sort().join(","); const v2 = [50 / 6 | 0, Math.round(-8.5), -73 % 4].join(","); const v3 = [NaN === NaN, null >= 0, "30" < "4"].map(Number).join(""); const v4 = "4" + 7 - 7 + "7"; console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
No node available, so traced by hand: default sort is lexicographic, Math.round(-8.5)=-8, -73%4=-1, false/true->0/1, string concat/coercion gives 407. Reasonably confident but could not execute to verify.
fix-1✓ pass1m 07s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1921 cents, but the correct quote is 2482: {"country":"JP","items":[{"grams":833,"qty":2,"price":1893,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 429, 862, 1173, 1832]; // cents, by zone const PER_STEP = [0, 86, 140, 187, 264]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5300, 8100, 18700, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"JP","items":[{"grams":582,"qty":5,"price":2948,"fragile":false}]} {"country":"FR","items":[{"grams":494,"qty":2,"price":1060,"fragile":false}]} {"country":"NZ","items":[{"grams":1761,"qty":1,"price":2377,"fragile":false},{"grams":1401,"qty":1,"price":2768,"fragile":false}],"coupon":"SHIP10"} {"country":"DE","items":[{"grams":157,"qty":1,"price":8384,"fragile":true},{"grams":690,"qty":2,"price":8034,"fragile":true}]} {"country":"DE","items":[{"grams":1326,"qty":1,"price":3218,"fragile":false},{"grams":307,"qty":3,"price":5149,"fragile":false},{"grams":229,"qty":1,"price":6788,"fragile":false}]} {"country":"FR","items":[{"grams":1720,"qty":5,"price":5653,"fragile":false}]} {"country":"BR","items":[{"grams":702,"qty":4,"price":4514,"fragile":true},{"grams":182,"qty":4,"price":5917,"fragile":false}],"coupon":"SHIP10"} {"country":"GB","items":[{"grams":1392,"qty":1,"price":8439,"fragile":false},{"grams":1289,"qty":2,"price":7572,"fragile":false},{"grams":918,"qty":1,"price":7588,"fragile":true},{"grams":615,"qty":1,"price":1856,"fragile":false}]} {"country":"US","items":[{"grams":1665,"qty":1,"price":1845,"fragile":false},{"grams":106,"qty":4,"price":8083,"fragile":true},{"grams":1122,"qty":5,"price":7871,"fragile":false}]} {"country":"DE","items":[{"grams":887,"qty":4,"price":2914,"fragile":false}]} {"country":"DE","items":[{"grams":765,"qty":4,"price":728,"fragile":false}]} {"country":"BR","items":[{"grams":660,"qty":5,"price":2512,"fragile":false}]} {"country":"JP","items":[{"grams":1450,"qty":4,"price":3893,"fragile":false},{"grams":1693,"qty":5,"price":7209,"fragile":true}]} {"country":"IT","items":[{"grams":1751,"qty":4,"price":3563,"fragile":true},{"grams":1155,"qty":5,"price":6232,"fragile":true},{"grams":1107,"qty":2,"price":6865,"fragile":true}],"express":true} {"country":"AU","items":[{"grams":1559,"qty":5,"price":6713,"fragile":false},{"grams":604,"qty":1,"price":4212,"fragile":false},{"grams":557,"qty":3,"price":880,"fragile":true}]} {"country":"FR","items":[{"grams":793,"qty":2,"price":1639,"fragile":false}]} {"country":"DE","items":[{"grams":106,"qty":1,"price":1858,"fragile":true}]} {"country":"IT","items":[{"grams":717,"qty":5,"price":1492,"fragile":false}]} {"country":"GB","items":[{"grams":359,"qty":5,"price":7443,"fragile":true}],"express":true} {"country":"BR","items":[{"grams":378,"qty":5,"price":6761,"fragile":false}],"coupon":"SHIP10"}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
Bug was grams ignoring qty; with grams += item.grams*item.qty the reported order quotes 2482 as expected. Ported to Python and ran all 20 orders.
implement-1✓ pass38s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[12,16],[16,23],[7,15],[25,32],[23,24]] [[27,27],[12,16],[5,6],[40,43]] [[38,46],[18,20],[25,26]] [[13,19],[35,43],[35,39],[22,23],[36,38],[15,23]] [[5,7],[32,38],[33,41],[25,30],[3,5],[28,34],[17,20],[28,28]] [[7,7],[29,37],[23,27],[31,35],[21,29],[20,20],[39,46]] [[17,25],[40,45],[10,11],[30,37],[38,46]] [[4,8],[32,37],[13,20],[30,36],[15,15],[14,14],[20,22]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Standard sort-and-merge with touching (s<=prev_end) merged. Ran on all 12 inputs.
repo-1✓ pass43s
prompt
Download airbench.ai/f/6c34230e648cde3b7bba50d8527d2bbc.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Bug was the overdraft condition `bal <= 0` instead of `bal < 0` (tests confirmed exactly-zero is not an overdraft). Fixed, all tests pass, sample prints the expected c7fa2b69.
repo-2✓ pass36s
prompt
Download airbench.ai/f/8beb5cbb6b67586e589a558bc7fe592a.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs: load sorted same-date txns by amount (should keep file order, stable sort by date only) and the overdraft condition bal<=0 instead of bal<0. Fixed both; tests pass and sample prints the expected a805b5fd.
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint nace-ai/Qwen3.8-Flash-Next-GGUF IQ4_XS @50e8661b (87 GiB, 3 shards; its card reports mean KL 0.063 from BF16 against Unsloth UD-IQ4_XS's 0.073 at the same size, a gap of -0.010 with a 95% CI of -0.013 to -0.008) on Unsloth's llama.cpp prebuild b11985-mix-c116c32 (linux-x64-cuda13-portable) in nvidia/cuda:13.3.1-runtime-ubuntu24.04 + libgomp1: llama-server -c 262144 -np 1 -fa on -ctk q8_0 -ctv q8_0 --jinja --mmproj mmproj-BF16.gguf, NO draft model, the rest llama.cpp's defaults (--fit places what fits in VRAM, the other experts on the CPU, the GGUF memory-mapped). The vision projector is Unsloth's BF16 mmproj for the same base model, since Nace publishes none and the vision axis cannot run without one; it is the untouched vision tower, not a quantized tensor. Served by the m64 capsule qwen38-flash-next-nace-iq4xs-5090 on one RTX 5090 with 64 GB of RAM, reached at the capsule container's port 9000. Decoding pinned greedy: checkup.py's proxy sets temperature 0, top_k 1, top_p 1, min_p 0 on every chat request. Harness: opencode 1.19.0 on the host: `opencode run --auto --format json <prompt>`; per-run opencode.json pins the model via @ai-sdk/openai-compatible; context 262144, max output 32768 tokens, image input declared only when the server passed a two-image probe. Orchestrator: github.com/dh7/agent-checkup-benchmark @ eca1c1c; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
Completed: every axis ran to the end inside its 120-minute budget, with no engine crash, harness timeout or operator kill. Scores by section: code 10/11, mail 6/6, store 4/4, text 9/9, vision 18/19 — 47/49 overall, 120 minutes wall clock. Nace side of the Flash-Next IQ4_XS head-to-head with Unsloth Dynamic v3.1 UD-IQ4_XS (checkup of 2026-11-11, 48/49 in 91 min). Identical engine (Unsloth llama.cpp b11985), context 262144, q8_0 KV, greedy decoding pinned by the harness proxy, and the same MTP draft head on both sides: Nace publishes none, so Unsloths
discussion
Sign in to join the discussion
No messages yet.