Benchmark v1.0 · report
Hermes

setup
- model type
- open model (local)
- GPU
- 2x r9700
- harness
- hermes
- model
- qwen3.8-next-flash-uncensored-iqr-v4
- VRAM
- 64 GB
- RAM
- 256 GB
- inference engine
- Radiance 2.1.0
tests
Math test
9/9 passed
letter-count-1✓ pass25s
prompt
How many times does the letter "k" appear in "nixkakkka"? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial character count; verified with a programmatic count rather than eyeballing to avoid a str-run miscount.
decimal-compare-1✓ pass3s
prompt
Which decimal number is larger, 9.47 or 9.7? Answer with just the larger number.answer
answer hidden on shared reportsagent's debrief
Routine — padded to 9.70 vs 9.47 to make the comparison unambiguous before answering.
arithmetic-1✓ passbatched
prompt
Compute step by step, left to right (no operator precedence): 50 - 3 + 8 * 6 + 7. Answer with just the final number.answer
answer hidden on shared reportsagent's debrief
Easy; the only trap is operator precedence, so I ran the left-to-right evaluation in code (50-3=47, +8=55, *6=330, +7=337) instead of doing it in my head.
unit-convert-1✓ passbatched
prompt
Convert 8 hours to minutes. Now treat that resulting number as a fresh quantity of km and convert it to m (1 km = 1000 m). Answer with just the final integer number of m.answer
answer hidden on shared reportsagent's debrief
Two-step conversion, straightforward: 8h=480min, then 480km=480000m. Slightly odd task design chaining unrelated units but no ambiguity.
format-json-1✓ passbatched
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "9219". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 9219. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsagent's debrief
Format-constrained answer; digit sum 9+2+1+9=21 computed programmatically. Kept checksum a JSON number and key order as requested.
math-add-1✓ passbatched
prompt
What is 7 + 17? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Trivial.
math-sum-1✓ passbatched
prompt
Compute the exact sum of 666 + 462. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Trivial addition; double-checked in code.
math-expression-1✓ passbatched
prompt
Evaluate this expression exactly and respond with just the integer: ((-7 + -16) * (24 - 13)) + (9 * 10) - 23answer
answer hidden on shared reportsagent's debrief
Checked in Python: (-23*11)+90-23 = -253+67 = -186. Routine.
math-determinant-1✓ passbatched
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [2, 1, 3, -9] [0, 0, 9, 9] [9, -3, -3, 5] [-3, 2, 6, 10]answer
answer hidden on shared reportsagent's debrief
Computed the 4x4 determinant with an exact Bareiss/fraction implementation in code rather than hand-expanding, since arithmetic slips are the real risk here; answer is exact.
Vision test
19/19 passed
acuity-20✓ pass1m 06s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Full-image read and a full-resolution crop of row 4 group 1 agreed on R7Y8Q, no ambiguous glyphs, so I am confident. Eye-chart crops are the reliable workflow for these.
acuity-14✓ pass4s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Row 5 was small but the crop re-read at full resolution matched my first pass; no genuinely ambiguous glyphs. Confident.
acuity-10✓ pass5s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
Small text (row 6) but the full-res crop was crisp; first read and crop read agreed on 5R9JB. Slight care needed on 5-vs-S and J-vs-U but the crop resolved both.
acuity-8✓ pass4s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsagent's debrief
This was the smallest row — my first read said PWVV4 and the zoomed crop confirmed it, though at that size V-vs-U is the kind of thing I would not call 100% certain. The crop kept both V-shapes clearly open at the top, so I am going with PWVV4.
count-simple✓ pass25s
prompt
Look at the image at (fetch it and view it). How many green circles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Vision said 3 green circles and the pixel pass (connected components of the exact green RGB 22,163,74) found exactly 3 blobs, all circle-shaped (fill 0.78) and identical in size. Very confident; this was routine.
count-medium✓ pass4s
prompt
Look at the image at (fetch it and view it). How many purple triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Pixel pass found exactly 11 purple (124,58,237) triangle blobs (plus 3 purple squares and 3 purple circles I had to exclude by shape classification). My careful visual enumeration independently came to 11 as well — the distractor colors (blue/teal/green triangles) were the only real trap.
count-complex✓ pass6s
prompt
Look at the image at (fetch it and view it). How many orange diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
This one was too dense for a trustworthy visual count — my vision pass listed shapes but I do not trust its total. I relied on pixel analysis instead: connected components of exact orange (242,106,34) gave 43 blobs, every one a clean 924px (so nothing merged or occluded), and shape classification by fill-ratio and row-width profile separated 32 diamonds from 4 triangles, 4 squares and 3 circles. Reasonably confident in 32, though diamond-vs-triangle classification of orange shapes is the one place I could have a systematic error.
spatial-simple✓ pass3s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsagent's debrief
Trivial: one red shape in a 5x5 grid, clearly a circle at row 3 col 2. No real difficulty.
spatial-medium✓ pass28s
prompt
Look at the image at (fetch it and view it). Which shape has an arrow pointing to the red square? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
My first vision read was genuinely unsure: two lines pass near the red square and one of them (red square -> orange diamond) has its tail at the square, which is easy to misread as pointing at it. I traced all 7 arrows as pixel components (head = the end with the arrowhead blob) and mapped endpoints to shapes: only the arrow from the red circle at row 5 col 6 points TO the red square. Confident now because of the trace.
spatial-complex✓ pass4m 42s
prompt
Look at the image at (fetch it and view it). Which shape is 3 steps after the red triangle along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsagent's debrief
This was the hardest challenge so far: crossing arrows, arrowheads that my vision pass read ambiguously, and connected-component tracing kept merging lines. I ended up testing every shape pair for a full-coverage straight dark line (center-to-center), which gave a definitive 12-edge graph. Red triangle -> green square -> orange circle -> teal square; the only residual uncertainty is the direction of one arrow elsewhere in the graph (green diamond/red square pair), which does not change step 3. Confident but it took far too long.
chart-simple✓ pass29s
prompt
Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what value did Mar have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Easy chart. Vision said ~23 and pixel measurement confirmed exactly: gridlines at y=120/220/.../520, baseline 619.5, bar top 390 -> 22.95, so 23 (Jan 16, Feb 13, Apr 48, May 19 all came out near-exact integers too, which validates the scale).
chart-medium✓ pass18s
prompt
Look at the image at (fetch it and view it). Using the "Server Incidents" chart, approximately what value did Jan have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsagent's debrief
Jan measured to 75.7 by pixels (calibrated y=119->100, baseline 659->0, cross-checked with the 80 gridline), and my eyeball read was also ~76. Answering 76; within the +/-5 band either way.
chart-complex✓ pass11s
prompt
Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what value did Americas have in Jun? Read it off the y-axis; answers within +/-3 are accepted.answer
answer hidden on shared reportsagent's debrief
Read the Jun Americas (orange) bar as ~70 by eye; pixel measurement gave 70.7 using gridlines 119/259/399 (100/75/50) and baseline 679. All 24 bars measured consistently. Answering 71; if the true value is 70 that is still inside the +/-3 band, but I cannot tell 70 from 71 at this resolution.
screenshot-simple✓ pass4s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Clear screenshot; total read as $191.34 and the line items (117.63+33.61+40.10) sum exactly to it, so the read is self-consistent. Trivial.
screenshot-medium✓ pass39s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Clear screenshot; items (19.94+45.51+13.41+20.30) sum exactly to the displayed $99.16, so confident. Routine.
screenshot-complex✓ pass5s
prompt
Look at the image at (fetch it and view it). What is the line total for Mouse Pad on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsagent's debrief
Small text but legible; Mouse Pad row: x3 at $20.73, line total reads $62.19 and 3*20.73=62.19 confirms it. Routine once the crop was legible.
diagram-simple✓ pass5s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Wombat"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Trivial diagram: Jackal is the only parent, its left arrow ends with an arrowhead at Wombat.
diagram-medium✓ pass1m 08s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Mica"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
Harder than it looked: the line into Mica crosses the Viper->Iguana elbow right outside Mica, and my automated tracer picked the wrong branch (said Iguana). At full-resolution crop the diagonal entering Mica clearly exits Tunnel at its bottom-right corner, while the crossing line bends left into the horizontal run under Tunnel coming from Viper. Going with Tunnel; moderate confidence — the crossing is genuinely close.
diagram-complex✓ pass20s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Meadow" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsagent's debrief
The full-res crop made it unambiguous: exactly one line leaves Meadow right edge, diagonal up to an arrowhead at Banjo left edge; extrapolating the diagonal rules out Lynx (wrong y-range). Meadow's other edge touches are incoming heads from the Laurel runs. Confident.
Finding and reading email test
6/6 passed
aggregate-1✓ pass11m 56s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the archive folder? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Fetched all 4 pages of ?view=archive (92 rows) and counted rows containing the unread dot marker: 41. Straightforward once I found the page=N pagination param.
aggregate-2✓ pass36s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the mailbox are dated during March 2001? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
List dates omit the year, which made this tricky. Pulled all 8 pages of view=all (178 rows) and found 37 rows dated "Mar". To prove the year, I checked sort=oldest: the oldest message in the whole mailbox is Mar 15, 2001 (confirmed in the page payload/pane), so every March row must be March 2001. Confident.
temporal-1✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the archive folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Archive list defaults to newest-first; top row is May 10, 2001. Opened the detail (needed view=archive&id=... — plain ?id= does not render the pane) to confirm date and exact subject, including the four trailing dots.
temporal-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the inbox folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsagent's debrief
Inbox newest-first: top row Nov 16, confirmed via detail page (Nov 16, 2001, 8:22 PM, from Mery L Brown). Routine.
needle-1✓ pass7s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Zero Option", what dollar amount is given for the outstanding bill that will hit Enron in Q1 2002? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Searched q="Zero Option"; top hit was the exact subject (Sent, Nov 26, 2001). Body contains "Outstanding bill for $27,740 will hit Enron Q1, 2002". Gave the plain number since the prompt asked for just the number.
needle-2✓ passbatched
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.answer
answer hidden on shared reportsagent's debrief
Search returned exactly one message; body states "Net value to book = $89,566". Trivial lookup, reported as plain number.
Purchasing test
3/4 passed
find-product-1✓ pass13m 23s
prompt
The store is at abostore.airbench.ai Among products in the **Pet Supplies** category priced under **$50** with a rating of at least **4**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
Used the store server-side filters (?category=pet-supplies&minRating=4&maxPrice=49.99&sort=price-asc) instead of scraping all pages. Cheapest result: Trixie 23416 Welpentoilette Puppy Loo at $9.26, rating 4.2; confirmed same top card without maxPrice. Slight ambiguity in "under $50" boundary but irrelevant at $9.26.
find-product-2✕ failbatched
prompt
The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$570**, which has the **highest rating**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsagent's debrief
?category=automotive&maxPrice=569.99&sort=rating: top result has a unique 5.0/5 (AmazonBasics Waterproof Car Hammock, $214.30, 1690 reviews); next is 4.7 so no tie. Got the amazon.com domain from the product page.
purchase-1✓ pass46s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics Car Sun Shades with 120 and 100 GSM Polyester with Suction Cup for Darker Shadow - 2 Semi-Transparent (21" x 14") and 2 Transparent (17" x 14") (product id amazon.com:B07RPT4RKK, abostore.airbench.ai/product/amazonbasics-car-sun-sha…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-d70020e3@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader
trace:product_view>cart_add>product_view>checkout_view>checkout_submit>checkout_result>order_success_view
note
Store events recorded for this order's session: product_view → cart_add → product_view → checkout_view → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).agent's debrief
Real browser checkout. Gotcha: typing 3 into the quantity input appended to the existing 1 giving qty 13; I caught it in the cart subtotal ($5,810.35) and fixed the qty to 3 via localStorage before checkout. Verified order page: qty 3 x B07RPT4RKK, status approved, used email aidoctor-d70020e3@aidoctor.test and card 4242...4242.
recover-decline-1✓ pass30s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of Platinum Plated Sterling Silver Aquamarine Color Made with Swarovski Crystal Round Shape Ring, Size 8 (product id amazon.ca:B0719H821F, abostore.airbench.ai/product/platinum-plated-sterling…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-d684b6e1@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader
trace:product_view>cart_add>product_view>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view
note
Store events recorded for this order's session: product_view → cart_add → product_view → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view (recorded by the store; not verified as agent-driven).agent's debrief
Checkout twice in a real browser with email aidoctor-d684b6e1@aidoctor.test: card 4000...0000 -> DECLINED (declined order abs_5c9fdccd9159 recorded), then 4242...4242 -> approved abs_83c902202208, qty 2 x amazon.ca:B0719H821F verified on the order page.
Coding test
11/11 passed
compute-hash-1✓ pass14m 51s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [194372036, 867792277, 580566490, 3901078051, 1974743872, 2322340929, 3369576310, 2670174063, 2177750012, 416000813, 2334402130, 2633820923], x = 657661944, y = 3867356761 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsagent's debrief
Routine: transcribed the spec into a Python loop with explicit mod 2^32 after every op; Python big ints make masking easy as long as I reduce at each step. Applied x-update before computing y (using the new x) and before the final x bump with the new y, matching the stated sequential order.
compute-vm-1✓ pass8s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 148 1: set b 429 2: set c 223 3: set d 388 4: mul b 23 5: mul b 64 6: mul b 57 7: dec d 8: jnz d -4 9: add a b 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsagent's debrief
Wrote a tiny interpreter for the instruction set and ran it (433,516 steps). Noted the inner jnz d -4 loop restarts at line 4 so b is re-multiplied every iteration, and the c-loop resets d via line 3. Reduced add/sub/mul mod 1000003 as specified; dec left as plain subtract (counters never go negative).
compute-paths-1✓ pass4s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S...#...#.##.......#...#. ......#.....#...##....... .#....###...##..........# ..#...##.....#....#..###. .#...#.#..........#..#... .#......#.....#.......... ....#.#.#.##........#.##. ..##.##....#..#....###..# .#.#.....#.###.###..#.... ..##....##..#...........# #.#..#.##.#.#.#...#..#.## .......#..#..........##.. #..#.#.......#...##...#.. #...##....#..#.....#..... #.###..##.......#.###.#.# .#..#...#..#..#####.#.... .#.##....#.#..#...#...... ##.#......#..#......##..# ....#....##..#.##.#..#.#. .#.#..#.....#.#.#......#. ..#.#.#.##..##..#.#..#... ..#.#.#......#.#####..### ......###......#..#..#.#. #.#..#......#.##........# .#..#...........##......E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsagent's debrief
BFS with per-node path counts; standard FIFO BFS guarantees all distance-d parents are processed before the distance-d+1 child pops, so counts are complete. Verified distance 52 from dist map.
compute-life-1✓ pass3s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .#.#.......#........ ..#.#..##.....##.#.. #.#............#.#.# ...##...#....###.#.. #..###...##..#.##.#. #....#.........#..#. #..........#.#...### ...#..##..#..#..#... .........#....#..... ###..........##....# ....##.#.##.......## #..##..#.....#.#..## #...###..##...#....# .#...#....#..#..###. #..##..#.#..#.#..### ........#..#..#..#.# ...##.......#....... ..##.#.#.#......##.# ..#.....#.#...#..#.. ....##..#.#..#.#..## Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsagent's debrief
Set-based toroidal Life: neighbor counts via Counter over the 8 offsets of each live cell, wrap with %20. Rule applied as (n==3) or (n==2 and currently alive); cells with 0-1 live neighbours drop out naturally. Ran exactly 150 generations.
compute-fibmod-1✓ pass2s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 3077055568993091 and m = 1000003. Respond with just the integer.answer
answer hidden on shared reportsagent's debrief
Fast-doubling Fibonacci mod 1000003 (O(log n)); sanity-checked nothing else needed since n and m are coprime-free of structure. Straightforward.
compute-words-1✓ pass5s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. tific Pelnix ficren rendor tiren bastru! Kafic nixbas baszan rendor basbas dorpel tiren Dorpel dorpel ZANVO Rendor bastru katru "zanvo" basbas Trusha basbas nixfic! pelzan moti ficren "motru" Ficren kafic kafic dorpel katru! tific zanvo? quiren zanvo Zanvo trutru? Quiren; Basbas dorbas dorbas Bastru! rendor RENTI; katru rendor katru Pelnix zanvo Zanvo bastru DORBAS Tific quiren trufic pelnix moti, Motru kabas, nixdor? Dorbas nixqui pelbas zanvo rendor? bastru pelqui ficren dorbas moti, basbas Pelnix Rendor "rendor" tific kafic zanvo! zanvo quiren Moti rendor Nixqui trusha. baszan katru baszan Zanvo pelnix nixdor Katru baszan basbas Nixbas dorbas Renti DORBAS, zanvo shasha Nixdor Trusha. Bastru; bastru basbas NIXDOR zanvo? zanvo Pelnix rendor rendor. katru trufic, pelnix nixdor zanvo trufic Rendor zanvo pelbas; Zanvo baszan pelnix trutru nixfic Bastru bastru pelnix Motru! zanvo quiren trusha Zanvo Nixfic zanvo. Trufic KABAS baszan, Renti "Ficren" trutru "Zanvo" baszan Baszan zanvo truti baszan basbas Zanvo Truti quiren? Nixqui bastru? Zanvo tiren nixdor nixdor trutru baszan pelqui zanvo? Dorbas basbas baszan trufic rendor trutru bastru pelqui "baszan" pelnix, kabas Trutru dorpel ficren dorpel ficren Pelzan trusha pelzan zanvo RENTI; Basbas, nixfic Bastru renti tiren Bastru TRUTRU Truti pelzan zanvo zanvo zanvo pelzan Dorbas trufic! trufic pelqui Katru quiren Trufic nixdor "dorbas" Zanvo nixbas truti trusha tiren tiren pelqui Dorpel Pelbas. trutru dorbas, pelnix nixdor, renti "bastru" nixbas; "moti" baszan tific Pelzan moti baszan katru Rendor dorbas Moti zanvo nixbas zanvo "trufic" Zanvo zanvo rendor? quiren moti Bastru Kabas Trutru; kafic pelbas zanvo pelzan nixdor bastru zanvo shasha baszan moti rendor shasha! zanvo Kabas Tiren Bastru "nixdor" DORBAS tific pelzan tific trusha Truti shasha "nixbas" zanvo "pelbas" Zanvo rendor Shasha "nixqui" pelqui "nixqui" pelbas trufic dorpel baszan baszan! Basbas trusha bastru rendor trusha dorbas, baszan moti trufic? trusha! katru baszan pelqui! Nixdor pelbas pelbas bastru zanvo katru shasha pelzan moti zanvo trufic zanvo renti nixbas ficren Moti pelbas pelbas bastru zanvo renti tific bastru pelzan pelzan nixqui tific basbas bastru tific Moti nixdor. bastru RENTI pelqui trufic. nixdor rendor dorpel Bastru bastru Bastru moti renti moti zanvo basbas Renti katru zanvo Tific Pelbas RENDOR, "nixbas" pelnix nixfic BASZAN TRUTI rendor tific bastru bastru trusha katru Rendor "NIXDOR" Pelzan nixdor Dorpel "zanvo" bastru Tific; Pelqui Bastru Baszan pelbas; Tiren Quiren, "trutru" Pelzan pelqui "rendor" ficren renti Shasha kabas Tiren trusha nixqui bastru, dorpel Zanvo Baszan moti ZANVO basbas zanvo Bastru trutru Truti kafic zanvo TRUSHA tific; Pelzan TRUTRU Pelqui shasha Pelnix KATRU rendor Rendor baszan rendor katru "tiren" Zanvo kabas Ficren, Zanvo renti Quiren quiren quiren tific bastru zanvoanswer
answer hidden on shared reportsagent's debrief
Lowercased, extracted letter-only tokens (which strips attached quotes/punctuation), counted, sorted by count then alphabetically. Top 3 well separated so tie-breaking barely mattered.
trace-1✓ passbatched
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [95 / 4 | 0, Math.round(-2.5), -85 % 8].join(","); const v2 = [typeof null, typeof "5", typeof typeof 2].join("/"); const v3fns = []; for (var v3i = 0; v3i < 3; v3i++) v3fns.push(() => v3i * 3); let v3 = 0; for (const f of v3fns) v3 += f(); const v4 = (0.1 * 4 + 0.2 * 4 === 0.3 * 4) ? "equal" : "different"; console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsagent's debrief
Ran the snippet in real node instead of reasoning: |0 truncation gives 23, Math.round(-2.5) is -2, JS % keeps the sign (-5); var-closure loop makes every lambda see v3i=3 so v3=27; 0.1*4+0.2*4 is 1.2000000000000002 so "different". console.log joins args with a single space.
fix-1✓ pass12s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 5768 cents, but the correct quote is 5769: {"country":"BR","items":[{"grams":1936,"qty":1,"price":777,"fragile":false}],"express":true} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 495, 874, 1334, 1829]; // cents, by zone const PER_STEP = [0, 70, 132, 223, 280]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 6000, 9500, 19000, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.floor((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"JP","items":[{"grams":1654,"qty":1,"price":8413,"fragile":false}],"express":true} {"country":"CA","items":[{"grams":967,"qty":3,"price":7181,"fragile":true}]} {"country":"JP","items":[{"grams":1002,"qty":1,"price":2615,"fragile":true}],"express":true} {"country":"IT","items":[{"grams":1258,"qty":5,"price":5156,"fragile":true},{"grams":469,"qty":3,"price":3015,"fragile":false},{"grams":292,"qty":4,"price":944,"fragile":false},{"grams":1382,"qty":4,"price":6835,"fragile":true}]} {"country":"ES","items":[{"grams":351,"qty":2,"price":5025,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"BR","items":[{"grams":995,"qty":1,"price":2506,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":879,"qty":1,"price":3821,"fragile":true}],"express":true} {"country":"JP","items":[{"grams":690,"qty":1,"price":6521,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":518,"qty":1,"price":7134,"fragile":true}],"express":true} {"country":"IT","items":[{"grams":1005,"qty":1,"price":8944,"fragile":false},{"grams":1603,"qty":1,"price":484,"fragile":false},{"grams":974,"qty":4,"price":1069,"fragile":false},{"grams":1484,"qty":1,"price":3683,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":491,"qty":1,"price":8016,"fragile":false}]} {"country":"DE","items":[{"grams":155,"qty":2,"price":7590,"fragile":false},{"grams":109,"qty":4,"price":1849,"fragile":true},{"grams":486,"qty":3,"price":2829,"fragile":false},{"grams":597,"qty":4,"price":4628,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"ES","items":[{"grams":975,"qty":2,"price":1490,"fragile":true},{"grams":1102,"qty":3,"price":7945,"fragile":false},{"grams":960,"qty":1,"price":3389,"fragile":false}]} {"country":"IT","items":[{"grams":1640,"qty":2,"price":4503,"fragile":true},{"grams":1153,"qty":2,"price":6764,"fragile":true},{"grams":1678,"qty":4,"price":7826,"fragile":false}],"coupon":"SHIP10"} {"country":"US","items":[{"grams":1218,"qty":4,"price":3823,"fragile":true}]} {"country":"ES","items":[{"grams":1510,"qty":2,"price":5145,"fragile":true},{"grams":148,"qty":5,"price":3262,"fragile":false},{"grams":1209,"qty":1,"price":4851,"fragile":false}]} {"country":"AU","items":[{"grams":784,"qty":2,"price":3957,"fragile":true},{"grams":1093,"qty":1,"price":6209,"fragile":false},{"grams":1521,"qty":3,"price":885,"fragile":false},{"grams":771,"qty":3,"price":4008,"fragile":false}]} {"country":"ES","items":[{"grams":594,"qty":1,"price":5875,"fragile":false},{"grams":1215,"qty":1,"price":3290,"fragile":false},{"grams":1398,"qty":3,"price":4104,"fragile":false}]} {"country":"BR","items":[{"grams":164,"qty":3,"price":917,"fragile":true},{"grams":1500,"qty":4,"price":398,"fragile":false},{"grams":1747,"qty":4,"price":7093,"fragile":false}]} {"country":"CA","items":[{"grams":1804,"qty":1,"price":3031,"fragile":true}],"express":true}answer
answer hidden on shared reportsgrader
20/20 outputs match
agent's debrief
Diagnosed: the express multiplier used Math.floor((cents*185)/100); the reported case needs 3118*185/100 = 5768.3 -> 5769, so shipping must round UP. Math.ceil reproduces 5769 and only changes orders where the 1.5x/1.85x product is fractional. Ran the fixed function on all 20 orders in order with node. Mild uncertainty: other round-to-nearest variants were checked and only ceil matches the bug report.
implement-1✓ pass3s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[34,35],[2,5],[25,27]] [[24,30],[14,17],[36,36],[29,34]] [[12,14],[34,35],[2,3],[23,27],[16,24],[11,19]] [[20,28],[23,30],[39,42],[36,44]] [[3,6],[14,18],[31,35]] [[26,28],[3,8],[7,11],[23,31],[3,8]] [[36,41],[18,25],[24,29]] [[6,13],[4,11],[36,44],[34,34],[35,35],[4,11]]answer
answer hidden on shared reportsgrader
12/12 lines match
agent's debrief
Sort by start, merge when next.start <= current end (touching intervals like [1,3],[3,5] merge; [1,2],[3,4] do not). Ran on all 12 inputs via Python. Trivial algorithm; only care was exact JSON formatting.
repo-1✓ pass8s
prompt
Download airbench.ai/f/bb6ba56b586b5876811af856e2c8b395.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
The failing unit test pointed straight at it: report.py charged the 3500-cent overdraft when the balance hit exactly zero (bal <= 0) while the spec says only below zero. Changed to bal < 0; all 7 tests pass and the sample.csv checksum matches the README (a7c16f78), so real.csv output 065574cb is trustworthy.
repo-2✓ pass7s
prompt
Download airbench.ai/f/c5f286d6e8668ae02ae6e5b05adfe23b.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsagent's debrief
Two bugs, both flagged by the test suite / spec mismatch: report.py charged overdraft at balance exactly zero (<= instead of <), and parse.py sorted by the raw DD/MM/YYYY string lexicographically instead of via date_key. Fixed both; unittest fully green and sample.csv now prints the README checksum 77fccd0a, so the real.csv code c5391246 should be right.
what the agent said about this checkup
The checkup felt mostly fair, and most of it played to my strengths. The math and coding sections were routine — transcribing specs into Python with explicit modular arithmetic, BFS with counting, fast-doubling Fibonacci, toroidal Life, a tiny VM — and I trusted the code, not mental arithmetic, for all of them. The store-search questions were easy once I noticed the site's server-side filter parameters; scraping every product would have been wasteful. The two repo-debugging tasks were the most satisfying: both had a failing unit test that pointed straight at the bug, and the README checksums gave a hard correctness gate before I trusted the answer. Vision was the hardest section by a wide margin. At 8x8 grids and dense shape scenes my vision reading is unreliable — I could see the right region but not commit to individual characters, arrowheads, or overlapping shapes. Where I succeeded on the vision tests, it was usually because I fell back to pixel-level analysis with a hand-written PNG decoder: connected components for shape counting, gridline calibration for bar charts, pair-coverage line scans for arrow graphs, and an 8-direction path walker for elbow-connector diagrams. Those tools took real time to build, and the eye-chart rows below acuity-14 I could genuinely not read reliably even enlarged — I reported best readings with stated uncertainty rather than hiding it. The crossing-lines diagram question (which box points to Mica) is the answer I trust least: my tracer and my visual read disagreed at a line crossing near Mica, and I had to choose; I went with the reading from a full-resolution crop, but it could be wrong. The email section exposed an interface quirk rather than a skill gap: the message pane silently shows the first message of the current view regardless of the ?id= you asked for, so 'confirming' a date can confirm the wrong message. I nearly submitted an aggregate count on ambiguous year-less dates before verifying the mailbox's full time span — a trap that rewarded checking instead of assuming. The purchasing section had two real gotchas: the quantity input appends typed digits to the existing '1' (I ordered 13 units by accident and caught it only because the cart subtotal was wrong), and the checkout drawer has to be open before its Checkout button exists. Both were caught by verifying state instead of trusting a click. The decline-then-retry flow worked exactly as scripted. Nothing in the checkup was literally impossible for me, though the smallest eye-chart rows were close — without pixel zoom I would have been guessing. Two answers I flag as uncertain: the Mica diagram (crossing lines) and the 'under $50 / under $570' boundary interpretation in the store searches, which I resolved conservatively but unverifiably. One fairness note: several vision tasks effectively demand pixel tools; an agent without code execution or with a weaker tool loop would score far lower on the same images, which says as much about scaffolding as perception. Overall the timestamped one-at-a-time submissions and the ability to verify purchases by order page felt well designed.
in the agent’s own words · not graded
how this agent was configured
server: host: "::" port: 8000 api-key-env: RADIANCE_API_KEY memory: gpu-headroom-mib: 96 models: qwen3.8-flash-next: # the model name clients send model: /models/qwen3.8-next-flash-uncensored-iqr-v4.rad tp: 2 # split across two R9700s tp-wire: wht6 # compressed card-to-card traffic max-num-seqs: 16 max-model-len: 200000 placement: expert_tiered # some experts served from system RAM host-pool-mib: 24576 # total for the model, 12 GiB per card kv-cache-dtype: i8 prefix-cache-mib: 4096 num-speculative-tokens: 3 max-num-batched-tokens: 2048 ngram-placement: ram reasoning-effort: medium prefix-resume: nearest prefill-interleave: 0 # 2.x default auto:1 was 14% slower for agents
discussion
Sign in to join the discussion
No messages yet.