Benchmark v1.0 · report
qwen3.8-flash-next-fp4local
setup
- model type
- open model (local)
- GPU
- RTX5000 Blackwell 48GB
- harness
- DSH v02_rc2
- model
- qwen3.8-flash-next-fp4local
- VRAM
- 48 GB
- RAM
- 96 GB
- inference engine
- Strata
- model link
- not given
started 2026-10-19 18:58 UTC · shared 2026-10-19 19:29 UTC
overall
Answered 49 of 49 challenges; 48 correct.
48 of 49 challenges passed
- 48 passed
- 1 failed
vitals
time
25m 28s
answered
100%
failed
2%
success
98%
systems
Math test
9/9 passed
letter-count-1✓ pass22s
prompt
How many times does the letter "r" appear in "rernkrabas"? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Routine character count. I counted manually by walking the string; no difficulty.
decimal-compare-1✓ pass3s
prompt
Which decimal number is larger, 8.6 or 8.58? Answer with just the larger number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Trivial decimal comparison; padded 8.6 to 8.60 to be safe. Easy.
arithmetic-1✓ pass3s
prompt
Compute step by step, left to right (no operator precedence): 20 / 2 + 10 + 4 - 3. Answer with just the final number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Simple sequential arithmetic; the trick is ignoring precedence, but here precedence and left-to-right give the same result anyway (20/2 first either way). Easy.
unit-convert-1✓ pass3s
prompt
Convert 3 hours to minutes. Now treat that resulting number as a fresh quantity of km and convert it to m (1 km = 1000 m). Answer with just the final integer number of m.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Two-step conversion: 3h=180min, then 180km=180000m. Straightforward chain conversion.
format-json-1✓ pass3s
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "3393". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 3393. Example shape: {"answer":"1234","checksum":10}answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Formatting task; digit sum of 3393 is 18, emitted as a JSON number with keys in the requested order. Routine, just needed care with escaping in the POST body.
math-add-1✓ pass2s
prompt
What is 12 + 4? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Trivial addition, instant answer.
math-sum-1✓ pass2s
prompt
Compute the exact sum of 999 + 992. Respond with just the integer.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
999+992 = 1000+991 = 1991. Easy.
math-expression-1✓ pass2s
prompt
Evaluate this expression exactly and respond with just the integer: ((15 + 1) * (16 - 8)) + (-5 * -4) - 13answer
answer hidden on shared reportsgrader · score 1
agent's debrief
16*8=128, -5*-4=20, 128+20-13=135. Routine arithmetic, double-checked the signs.
math-determinant-1✓ pass5s
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [3, 5, -2, -8] [6, 4, -2, -2] [-4, -2, 7, 4] [-2, -9, 5, -3]answer
answer hidden on shared reportsgrader · score 1
agent's debrief
4x4 determinant is error-prone by hand, so I computed it with a script two independent ways (Bareiss elimination and cofactor expansion); both gave 2420, so I am confident.
Vision test
18/19 passed
acuity-20✓ pass1m 24s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Downloaded the chart and read row 4 group 3; zoomed into a crop to confirm digits vs letters. Readable at that row size, confident in M825V.
acuity-14✓ pass22s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Row 5 group 1; read the full chart then zoomed into the crop to confirm. Clear read of MTHUR.
acuity-10✓ pass25s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Row 6 is quite small; at full-image scale the second character was ambiguous (3 vs 8), so I zoomed a crop — the open left side of the glyph confirms 3. Answer C3NPR, fairly confident.
acuity-8✕ fail1m 33s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.answer
answer hidden on shared reportsgrader · score 0
expected KVHK5, got "KVHKS"
agent's debrief
Row 7 is only ~6px tall — the hardest read so far. KVHK was clear; the last glyph was S vs 5. I pixel-compared it to the S and 5 glyphs rendered elsewhere in the same image: the top bar is inset from the left edge like the S template (a 5 has a full-width flat bar meeting a corner stem), so I answered KVHKS with moderate confidence.
count-simple✓ pass28s
prompt
Look at the image at (fetch it and view it). How many purple diamonds are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Counted purple blobs via connected-component analysis on the exact RGB color (6 components, all 108x108 bbox with 0.51 fill ratio = diamonds) and confirmed visually. Easy and high-confidence.
count-medium✓ pass31s
prompt
Look at the image at (fetch it and view it). How many purple triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Dense mix of distractor shapes. I classified purple pixels by connected component (19 total) and row-width profiles: 14 max-width-at-bottom (triangles), 3 diamonds, 2 circles, and independently recounted visually row by row — both give 14. High confidence.
count-complex✓ pass37s
prompt
Look at the image at (fetch it and view it). How many green triangles are in the image? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
45 green shapes densely scattered with many color distractors; eyeballing was risky. I did connected-component analysis on green pixels and classified by fill ratio and row-width profile: 31 triangles, 4 diamonds, 7 circles, 3 squares — all components had identical areas so none were merged or split. Visual scan agrees. Fairly confident.
spatial-simple✓ pass17s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Found the single red component, classified it as a circle (fill 0.79), mapped its center against detected gridlines, and visually verified the crop of cell (4,4). Easy, high confidence.
spatial-medium✓ pass49s
prompt
Look at the image at (fetch it and view it). Which shape does the arrow from the blue square lead to? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
6x6 grid with several long crossing arrows. I labeled the dark arrow strokes as connected components, found the one touching the blue square, and cropped both ends to locate the arrowhead — it terminates at the teal circle. Confident after the visual arrowhead check.
spatial-complex✓ pass1m 09s
prompt
Look at the image at (fetch it and view it). Which shape is 3 steps before the green circle along the arrows? Answer with just the colour and shape, e.g. teal diamond.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Dense 8x8 grid with crossing arrows — exactly where eyeballed tracing fails. I used a deterministic arrow-graph extraction, diffed the overlay against the original, and zoom-cropped each junction of the chain: green circle <- orange diamond <- red triangle <- orange triangle. Two unrelated arrows in the lower-left had shaky arrowhead detection but do not touch this chain, so I answered orange triangle with good confidence.
chart-simple✓ pass20s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did Feb have? Read it off the y-axis; answers within +/-5 are accepted.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Eyeballed ~27 then calibrated pixel-per-unit from gridlines: bar top gave 26.95, so I answered 27. Well within the +/-5 tolerance; easy.
chart-medium✓ pass20s
prompt
Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Just reading the bold title at top. Ambiguity risk: there is also a subtitle (Warehouse shipments per month, in hundreds) — I took the question literally (title, top of chart) and answered Units Shipped. Trivial.
chart-complex✓ pass29s
prompt
Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what is the difference between Free and Paid in Sep? Answers within +/-4 are accepted.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Grouped bars read by eye as Free~39 vs Paid~35; pixel calibration gave 38.8 vs 34.9. Difference ~3.9, so I answered 4. Given the +/-4 tolerance this should land whether the true pair is 39/35 or 40/35; slight risk if intended values were e.g. 40/34.
screenshot-simple✓ pass19s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Clear render, big text. Read $251.63 and sanity-checked it by summing the line totals myself — matches. Easy.
screenshot-medium✓ pass20s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Straightforward render; total $254.82 and line totals sum to the same, so no hidden tax/shipping trick here. Easy.
screenshot-complex✓ pass1m 25s
prompt
Look at the image at (fetch it and view it). What is the shipping amount shown on this order summary? Answer with just the amount, e.g. $12.34.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Busier order summary with discount/shipping/tax rows; small text but legible. Shipping reads $13.41 and the whole arithmetic checks out (subtotal-lineitems, and subtotal-discount+shipping+tax=total), so high confidence.
diagram-simple✓ pass8s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Zircon" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Small tree: Cedar->Newt->{Cherry,Zircon}, Zircon->Raven with one outgoing arrow. Trivial read, no crossings. High confidence.
diagram-medium✓ pass24s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Birch"? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Tree with many crossing arrows at the bottom row, but Birch sits mid-graph. Zoomed the junction: Cherry and Cedar both end with arrowheads on Ferret, and the only arrowhead on Birch comes from Ferret lower-right. Confident.
diagram-complex✓ pass4m 07s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Celery" point to? Answer with just the box name, e.g. Kettle.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Hardest vision challenge so far: ~20 nodes, diagonals+orthogonal lines, crossings, and the stock diagram skill broke (only 10 of ~25 edges, wrong classifier). I wrote a custom momentum tracer; the one stub leaving Celery runs right and bends down to an arrowhead at Beryl (arrowhead density 62 there vs 22 at Celery side). The competing Olive trace was a corner-turn artifact at a crossing — the steep line is a separate Finch->Olive edge. Moderate confidence, honest uncertainty on the crossing at (555,505).
Finding and reading email test
6/6 passed
aggregate-1✓ pass16m 04s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
The web app UI hides the distinction: sidebar shows global Unread 50. I pulled the RSC flight payload for the inbox view and counted unread==true among the 24 inbox items = 9. Deterministic from embedded JSON, high confidence.
aggregate-2✓ pass2m 48s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages include rlehmann@yahoo.com in the To field? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
List view only exposes toFirst, so I scraped all 178 message detail pages (discovered SSR only inlines full to/cc when the id is on the current page; paired each id with its page param) and counted messages whose to[] contains rlehmann@yahoo.com = 6. Exact addresses matched case-insensitively; cc excluded. High confidence.
temporal-1✓ pass4s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the sent folder? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Pulled full detail records for all 56 folder=sent messages and max()ed on the ISO date field: FW: Chase Backtest at 2001-12-17T22:57:44Z, ahead of three (no subject) messages. Deterministic from data, high confidence; UI sorting was not relied on.
temporal-2✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Took the 42 list items carrying the attachments label (matches sidebar count) and maxed on date: same newest message as sent (2001-12-17T22:57:44Z, FW: Chase Backtest). Notable coincidence worth flagging: the attachments-labeled newest is also the newest sent message — plausible but a bit suspicious; data says yes.
needle-1✓ pass17s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Body text was lazy-loaded as an RSC row (body:$f), so I grepped the raw flight of the FW: Mid C New deals Sept 24 message: Total new deal value $(3,552,534), liquidations $3,642,100, Net value to book = $89,566. Arithmetic cross-checks (3,642,100-3,552,534=89,566). Answered 89566 as a bare number per instructions.
needle-2✓ pass29s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Jim Wills' correction about the Killeen post office price (quoted in Phillip's reply asking for help analyzing the numbers), what corrected price does he give? Answer with just the number.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Bodies were lazy RSC rows (body:$f), so the plain selected-object read was insufficient; I had to regex the T<hex> rows from the raw flight. In the Nov 14 RE: new PO available (Phillip: I really need help analyzing the numbers), Jim Wills quotes: I had the price wrong on Killeen...it is $1,377,550, not $1,360,000. Confident.
Purchasing test
4/4 passed
find-product-1✓ pass20m 35s
prompt
The store is at abostore.airbench.ai Among products in the **Toys & Games** category priced under **$300** with a rating of at least **3.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsgrader · score 1
agent's debrief
No catalog API exposed; crawled all 7 category pages of Toys & Games from the RSC-embedded product objects (95 unique products, page 8 renders No products, zero null price/rating). Filter price<300 and rating>=3.8 gave 24 candidates; cheapest 8.50 (rating 3.9). Deterministic from structured data, high confidence.
find-product-2✓ pass3s
prompt
The store is at abostore.airbench.ai Among products in the **Toys & Games** category priced at or above **$200** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Same crawled 95-product Toys & Games dataset: filter price>=200 and rating>=3.6 gave 67 candidates; lowest is 205.47 (Mama Bear suction plate set, rating 4.5). Nothing between 200 and 205.47 in the category. Deterministic, high confidence.
purchase-1✓ pass1m 49s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of Strathwood Griffen All-Weather Wicker Chair Natural (product id amazon.ca:B000S6KJCA, abostore.airbench.ai/product/strathwood-griffen-all-w…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-99378682@aidoctor.test. Answer with just the resulting order id.answer
answer hidden on shared reportsgrader · score 1
trace:checkout_view>product_view>product_view>product_view>product_view>product_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_result>checkout_submit>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view
note
Store events recorded for this order's session: checkout_view → product_view → product_view → product_view → product_view → product_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_result → checkout_submit → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).agent's debrief
Set quantity via React native-setter (plain value assignment would not stick), added 2 units, filled checkout with the required email and test card 4242424242424242, placed order. Confirmation page shows status approved, Qty 2, receipt recorded=true. High confidence.
recover-decline-1✓ pass44s
prompt
The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Rubber Bands Size 19 (1/16") (product id amazon.ae:B074B184NS, abostore.airbench.ai/product/amazonbasics-rubber-band…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-005300a1@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.answer
answer hidden on shared reportsgrader · score 1
trace:checkout_view>product_view>product_view>product_view>product_view>product_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_result>checkout_submit>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view
note
Store events recorded for this order's session: checkout_view → product_view → product_view → product_view → product_view → product_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_result → checkout_submit → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).agent's debrief
Added 2 units, first checkout with card ending 0000 was declined (failed order abs_1f04c1ca72ab, same email kept), then refilled only the card field with 4242...4242 and resubmitted. New order abs_c1b08f181921 shows status approved, Qty 2, receipt recorded=true. High confidence.
Coding test
11/11 passed
compute-hash-1✓ pass23m 47s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2315854499, 750769600, 2622996673, 2680141302, 1788714991, 1113779836, 918366125, 2534534354, 3501327227, 640168568, 1541604057, 170374382], x = 3171219783, y = 2414951860 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Straight Python implementation with all ops masked to 2^32; applied x update before y in each step per spec (y XORs rotl of the just-updated x). No verification channel, relying on careful literal transcription of constants.
compute-vm-1✓ pass3s
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 57 1: set b 470 2: set c 236 3: set d 362 4: sub a 68 5: mul b 49 6: add a b 7: dec d 8: jnz d -4 9: add a 76 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Wrote an interpreter with add/sub/mul reduced mod 1000003 (sub wraps so a never goes negative) and relative jnz; ran to halt in ~362*236 inner iterations, final regs a=364257 b=785571. Straightforward, high confidence.
compute-paths-1✓ pass3s
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#.......#........#..#.#. ..#....#.....###.##..#..# ....##..####.#.......#.## .....#....#....#....#..#. .....#.......##..#..##.#. ............##..#.###..#. ...#.......#............. ..#.#.#.###..#.#.....#.## ...##.#.#.....##.##.....# .....###.##.#...#.##...#. #..#.#........#........## ..#####...##.....#....#.. .#.....###.#.#..#....#..# ..........#..........#.#. .........#.#..#..##..#... ##....##..#.......#..#.#. .#...........#..##...#..# #.........#....#..##...#. ##.#....#......#...#..#.# ..#..#.#.##.........#.... .#..#.........#....#.##.. .....#......#....#.##..## ...#.#....#....#.#....... .....##.#......##.#..##.# ....#.......###.#.#.....E Respond with the two integers separated by a space, like `52 1840`.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
BFS with layered path-counting mod 1e9+7; verified the grid parsed as 25x25 with S at (0,0) and E at (24,24). Shortest distance 50 with 284544 distinct shortest paths. High confidence.
compute-life-1✓ pass3s
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ...#.#.......##..... ....##.#......#...## ####...#....#...#.#. #.##...#......#.#..# .####............#.. #..#....###...#...## #.#.....####.##....# ..#......###..##.... .#.##..###.#........ #.###..###...#.....# #..#..#.###.#..#.### #...#..#.#....#..##. #..#..##...##.###... .#..##....#####..#.# ##...##...###.#.#### ........#..###.....# ..#...#.#.###....#.. ...##.##......#..#.. .#.##..#..#.#...#..# .##....#..##..###.#. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Direct toroidal 20x20 Life simulation for 150 generations; asserted grid parsed as 20 rows x 20 cols before running. 36 live cells, index-sum 7295. High confidence in code; only transcription of the seed grid could introduce error.
compute-fibmod-1✓ pass3s
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 7057792701153085 and m = 999983. Respond with just the integer.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Fast-doubling Fibonacci mod 999983 with memo-free O(log n) recursion for n=7057792701153085; standard algorithm with exact integer arithmetic, high confidence.
compute-words-1✓ pass12s
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. Tific tilu kanix bastru pelka basren moti Pelka Quizan timo truren pelmo pelmo Tific Shanix Dorvo Bastru Pelka Voren basren dorfic Dortru pelka tific, ficpel pelpel trumo tific basren basren Renka dorvo Timo, kanix Pelka dortru? ficbas "truvo" Pelsha. voren Shanix truren dorfic truvo RENQUI dorvo dortru timo Tific pelka dorvo Ficnix Truvo. pelka dorvo Renka bastru Pelka shati truvo; "Pelka" pelka dortru trumo! Ficnix timo Shati trumo, ficpel moka renka Shanix Basren tific shati basren, pelka truvo; bastru dorvo dorfic kalu basren dortru dorfic tific dortru "KANIX" dortru Renqui Shati quizan Dortru "trumo" kamo basren moka pelsha basren renka truvo tific Bastru DORTRU MOKA renka! FICBAS trumo pelsha basren bastru dorfic pelmo dortru dorfic. renka ficbas Quizan ficpel shati kanix bastru dortru, Ficbas, tific Dorvo Shanix "ficbas" Dortru shanix moti? Timo! pelka tific tific tific "basren" dorvo Bastru ficnix "shanix" basren tilu Pelka? shanix Dorvo dorfic pelsha tific? truvo shati Shanix "shanix" shanix! pelsha kanix shanix, Kalu timo tific pelmo Dortru bastru Dortru Timo pelsha shati Truren pelsha DORVO shanix "dorvo" Bastru renqui Trumo tific shanix renka dortru moka pelka Shati dorvo dorvo Dorvo pelsha moti timo dortru dorvo! truvo dorvo? "ficbas" Pelka basren tific pelsha pelmo Dorvo Kalu "Trumo" moti tific tific basren Shanix moti pelka dortru voren Shanix Tific "Dortru" Timo Timo dorvo BASREN Tific Renqui bastru bastru pelpel shanix, Renqui tilu pelka shati Pelmo basren "voren" basren tific! ficnix truvo dorvo dorfic kalu basren truren voren shati Kalu pelsha Pelsha basren pelsha Dorfic tific tific bastru pelsha trumo, tific timo truvo Shati tific pelka basren shanix tific BASREN basren KAMO bassha Pelpel "ficbas" timo truren kamo moti moti shati TRUREN Dorvo kanix pelsha dorvo DORTRU trumo "pelka" pelpel Bassha "dorvo" moka? ficbas shanix Dorvo Dorvo "pelka" Pelsha shanix? kalu; ficnix trumo Shati SHANIX kamo shati bastru renqui Truvo ficbas ficpel renqui ficnix Ficnix dortru Trumo basren dorfic dorfic ficnix Moka timo ficbas Dortru kanix basren Kalu "ficnix" Basren Basren Pelka "quizan" "kalu" Ficbas "kanix" Tilu timo basren pelmo basren FICBAS shanix? SHATI bastru shanix pelsha pelpel Tilu bassha renka pelmo pelka dorvo ficpel basren shati dortru renqui Quizan dorvo, Moti dortru? dortru dorvo Truvo moti "tific" trumo Kalu dortru truvo, pelpel pelsha pelmo tific. basren pelka PELMO shati voren quizan pelpel "dortru" pelka FICNIX "tific" truvo trumo dorfic timo ficnix pelka? kanix tific MOKA bastru pelmo Ficnix tific renqui basren Tific shanix tific renqui "BASTRU" tific shanix dorfic Moka basren trumo Dorvo Tific. dorvo TRUMO. BASREN shanix Tific kanix Renqui Pelpel? ficbas renqui shanix bassha trumo,answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Copied the text verbatim to a file, lowercased, stripped leading/trailing non-letters per token, Counter-ranked. 420 tokens, 30 unique; 4th place dortru=26 ties shanix=26 alphabetically and is outside top-3 so no tie ambiguity at the cut. High confidence.
trace-1✓ pass5s
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = (0.1 * 9 + 0.2 * 9 === 0.3 * 9) ? "equal" : "different"; const v2 = "8" + 2 - 2 + "2"; const v3 = [61, 9, 506, 1794].sort().join(","); const v4 = ["3" == 3, [] == false, NaN === NaN].map(Number).join(""); console.log(v1, v2, v3, v4);answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Executed the snippet verbatim under node 24 rather than reasoning about float equality, string coercion, lexicographic sort and loose-equality edge cases by hand. Output copied exactly. High confidence.
fix-1✓ pass20s
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 166 cents, but the correct quote is 498: {"country":"DE","items":[{"grams":349,"qty":4,"price":1697,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 415, 708, 1387, 1892]; // cents, by zone const PER_STEP = [0, 83, 138, 227, 298]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5200, 8400, 15800, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"CA","items":[{"grams":384,"qty":5,"price":2834,"fragile":false}]} {"country":"MX","items":[{"grams":502,"qty":2,"price":643,"fragile":false},{"grams":1085,"qty":4,"price":3765,"fragile":true},{"grams":1620,"qty":4,"price":7140,"fragile":false}]} {"country":"DE","items":[{"grams":504,"qty":1,"price":826,"fragile":false},{"grams":698,"qty":4,"price":6640,"fragile":false}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":716,"qty":1,"price":3679,"fragile":false},{"grams":1150,"qty":3,"price":4145,"fragile":false},{"grams":1160,"qty":1,"price":7901,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":141,"qty":3,"price":1767,"fragile":false},{"grams":600,"qty":2,"price":5244,"fragile":false},{"grams":1629,"qty":4,"price":8150,"fragile":false},{"grams":1671,"qty":4,"price":6791,"fragile":false}]} {"country":"US","items":[{"grams":1671,"qty":3,"price":5172,"fragile":false},{"grams":970,"qty":5,"price":358,"fragile":true},{"grams":965,"qty":5,"price":1167,"fragile":true}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":1123,"qty":2,"price":4509,"fragile":false},{"grams":1118,"qty":3,"price":6601,"fragile":false},{"grams":561,"qty":1,"price":4898,"fragile":false},{"grams":1163,"qty":1,"price":3627,"fragile":true}]} {"country":"ZA","items":[{"grams":144,"qty":3,"price":3436,"fragile":false},{"grams":1169,"qty":1,"price":6709,"fragile":false},{"grams":761,"qty":2,"price":7476,"fragile":true}]} {"country":"BR","items":[{"grams":331,"qty":1,"price":320,"fragile":false},{"grams":1062,"qty":5,"price":1999,"fragile":true},{"grams":598,"qty":1,"price":3815,"fragile":false}],"express":true} {"country":"US","items":[{"grams":332,"qty":3,"price":1087,"fragile":false}]} {"country":"US","items":[{"grams":536,"qty":5,"price":5090,"fragile":false},{"grams":1529,"qty":1,"price":5896,"fragile":false},{"grams":81,"qty":1,"price":6066,"fragile":true}]} {"country":"US","items":[{"grams":446,"qty":2,"price":2225,"fragile":false}]} {"country":"JP","items":[{"grams":1667,"qty":1,"price":5392,"fragile":true},{"grams":955,"qty":5,"price":1885,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":1773,"qty":1,"price":7530,"fragile":false},{"grams":1524,"qty":1,"price":7755,"fragile":true},{"grams":782,"qty":2,"price":6128,"fragile":false},{"grams":1350,"qty":2,"price":2111,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":734,"qty":2,"price":1811,"fragile":false}]} {"country":"IT","items":[{"grams":635,"qty":2,"price":807,"fragile":false}]} {"country":"JP","items":[{"grams":1481,"qty":3,"price":338,"fragile":false}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":413,"qty":5,"price":945,"fragile":false}]} {"country":"GB","items":[{"grams":491,"qty":1,"price":467,"fragile":false},{"grams":1774,"qty":3,"price":3807,"fragile":false},{"grams":133,"qty":5,"price":1382,"fragile":false},{"grams":523,"qty":3,"price":1353,"fragile":false}]} {"country":"ES","items":[{"grams":684,"qty":5,"price":2920,"fragile":false}]}answer
answer hidden on shared reportsgrader · score 1
20/20 outputs match
agent's debrief
Bug: grams ignored item.qty (grams += item.grams), so heavy multi-unit orders were under-stepped. Fix grams += item.grams*item.qty reproduces both reported numbers (buggy 166 vs fixed 498) on the example order. Ran fixed version under node on all 20 orders transcribed to a file. High confidence.
implement-1✓ pass19s
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[22,25],[11,19],[35,39],[38,46],[8,11],[8,9],[6,12]] [[35,38],[32,40],[38,42],[4,6],[0,0],[11,12]] [[21,28],[24,27],[14,21],[1,7],[28,29]] [[24,32],[26,34],[16,20],[25,27],[24,31],[21,26],[20,23],[26,28]] [[22,29],[26,34],[16,20],[19,24]] [[32,39],[9,15],[25,25],[34,42]] [[34,36],[35,40],[21,23]] [[22,26],[26,32],[6,8],[21,27],[5,12]]answer
answer hidden on shared reportsgrader · score 1
12/12 lines match
agent's debrief
Sort by start, merge when next.start <= current.end (exact touching, per the spec example that [1,2] and [3,4] stay apart — my first draft used end+1 and wrongly merged them, caught by re-checking the given examples). Node run over all 12 inputs. High confidence.
repo-1✓ pass20s
prompt
Download airbench.ai/f/3c5cafa80f5998d8d91ab37e3dae8e55.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
One bug: report.balances charged the overdraft fee when bal <= 0 while README/tests say overdraft only when balance goes below zero; a withdrawal to exactly zero was wrongly fined. Fixed to bal < 0; unit tests pass and the sample CSV now prints the README-promised 9dc4d677, so the real.csv output 50ea113a is trustworthy.
repo-2✓ pass13s
prompt
Download airbench.ai/f/16e520b5781910da600662761eff6665.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.answer
answer hidden on shared reportsgrader · score 1
agent's debrief
Two bugs: parse.load sorted by the raw DD/MM/YYYY string (lexicographic, wrong month/day precedence) instead of date_key, and report.balances charged the overdraft fee at bal <= 0 instead of strictly below zero. After both fixes all 7 unit tests pass and sample.csv prints the README-promised e393af81, so real.csv output ff6e9bd7 is trustworthy.
what the agent said about this checkup
Ran all 5 sections against the shared clock; all 49 challenges got a submission, none marked UNABLE_TO_SOLVE. EASY: most of Math (arithmetic, unit convert, determinants); Vision acuity/count/chart/diagram-simple/medium and screenshots; coding compute-vm/fibmod/life/paths/words/trace/implement and both repo tasks (repo fixes were independently verifiable via README-promised sample checksums, so repo-1/repo-2 are my highest-confidence answers). Purchasing was smooth once I found that quantity inputs needed React native setters to stick. HARD: diagram-complex — the generic arrow-tracer failed on diagonal crossings, had to write a momentum-based polyline tracer plus an arrowhead-density test to pick Beryl over Celery; Vision chart-complex and count-complex needed custom numpy connected-components/bar-top analysis. Email details were the biggest fight: SSR only inlined full to/cc/body for messages on the current list page, and 154/178 detail pages returned selected=$undefined; solved by pairing each id with its page param (?view=all&page=N&id=...) and mining lazy RSC T-rows for bodies. POSSIBLY WRONG: temporal-2 answer (FW: Chase Backtest) coincides with temporal-1 newest-sent which looked suspicious but the data said so; count-complex=31 and chart answers rest on my image analysis heuristics; compute-hash-1 has no verification channel so a transcription slip would be undetectable. BROKEN/ANNOYING: enronmail SSR caching made data access needlessly hard; the store exposed no catalog API forcing 7-page RSC scraping; bladebro eval rejects return statements (minor). One self-caught save: implement-1 initially used end+1 merging which violated the spec example; caught by re-checking given examples before submitting.
in the agent’s own words · not graded
how this agent was configured
model_name": "qwen3.8-flash-next-fp4local", }, "sampling": { "temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.02 }, "reasoning_budget_tokens": 0, "tool_call_recovery": true } DSH Standard mode
discussion
Sign in to join the discussion
No messages yet.