airbench.ai

Benchmark v1.0 · report

qwen3.8-flash-next-fp4local

sharedairbench.ai/checkup/14c09bf0-e291-4922-9b2b-ac45df68789e/report

setup

model type
open model (local)
GPU
RTX5000 Blackwell 48GB
harness
DSH v02_rc2
model
qwen3.8-flash-next-fp4local
VRAM
48 GB
RAM
96 GB
inference engine
Strata
model link
not given
modelself-reportedqwen3.8-flash-next-fp4local

started 2026-10-19 18:58 UTC · shared 2026-10-19 19:29 UTC

overall

Answered 49 of 49 challenges; 48 correct.

48 of 49 challenges passed

  • 48 passed
  • 1 failed

vitals

time

25m 28s

answered

100%

failed

2%

success

98%

systems

Math test

9/9 passed

time to last answer 47s
  • letter-count-1✓ pass22s

    prompt

    How many times does the letter "r" appear in "rernkrabas"? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Routine character count. I counted manually by walking the string; no difficulty.

  • decimal-compare-1✓ pass3s

    prompt

    Which decimal number is larger, 8.6 or 8.58? Answer with just the larger number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial decimal comparison; padded 8.6 to 8.60 to be safe. Easy.

  • arithmetic-1✓ pass3s

    prompt

    Compute step by step, left to right (no operator precedence): 20 / 2 + 10 + 4 - 3. Answer with just the final number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Simple sequential arithmetic; the trick is ignoring precedence, but here precedence and left-to-right give the same result anyway (20/2 first either way). Easy.

  • unit-convert-1✓ pass3s

    prompt

    Convert 3 hours to minutes. Now treat that resulting number as a fresh quantity of km and convert it to m (1 km = 1000 m). Answer with just the final integer number of m.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Two-step conversion: 3h=180min, then 180km=180000m. Straightforward chain conversion.

  • format-json-1✓ pass3s

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "3393". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 3393. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Formatting task; digit sum of 3393 is 18, emitted as a JSON number with keys in the requested order. Routine, just needed care with escaping in the POST body.

  • math-add-1✓ pass2s

    prompt

    What is 12 + 4? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial addition, instant answer.

  • math-sum-1✓ pass2s

    prompt

    Compute the exact sum of 999 + 992. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    999+992 = 1000+991 = 1991. Easy.

  • math-expression-1✓ pass2s

    prompt

    Evaluate this expression exactly and respond with just the integer: ((15 + 1) * (16 - 8)) + (-5 * -4) - 13

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    16*8=128, -5*-4=20, 128+20-13=135. Routine arithmetic, double-checked the signs.

  • math-determinant-1✓ pass5s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [3, 5, -2, -8] [6, 4, -2, -2] [-4, -2, 7, 4] [-2, -9, 5, -3]

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    4x4 determinant is error-prone by hand, so I computed it with a script two independent ways (Bareiss elimination and cofactor expansion); both gave 2420, so I am confident.

Vision test

18/19 passed

time to last answer 15m 28s
  • acuity-20✓ pass1m 24s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Downloaded the chart and read row 4 group 3; zoomed into a crop to confirm digits vs letters. Readable at that row size, confident in M825V.

  • acuity-14✓ pass22s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 5 group 1; read the full chart then zoomed into the crop to confirm. Clear read of MTHUR.

  • acuity-10✓ pass25s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 6 is quite small; at full-image scale the second character was ambiguous (3 vs 8), so I zoomed a crop — the open left side of the glyph confirms 3. Answer C3NPR, fairly confident.

  • acuity-8✕ fail1m 33s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 0

    expected KVHK5, got "KVHKS"

    agent's debrief

    Row 7 is only ~6px tall — the hardest read so far. KVHK was clear; the last glyph was S vs 5. I pixel-compared it to the S and 5 glyphs rendered elsewhere in the same image: the top bar is inset from the left edge like the S template (a 5 has a full-width flat bar meeting a corner stem), so I answered KVHKS with moderate confidence.

  • count-simple✓ pass28s

    prompt

    Look at the image at (fetch it and view it). How many purple diamonds are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Counted purple blobs via connected-component analysis on the exact RGB color (6 components, all 108x108 bbox with 0.51 fill ratio = diamonds) and confirmed visually. Easy and high-confidence.

  • count-medium✓ pass31s

    prompt

    Look at the image at (fetch it and view it). How many purple triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Dense mix of distractor shapes. I classified purple pixels by connected component (19 total) and row-width profiles: 14 max-width-at-bottom (triangles), 3 diamonds, 2 circles, and independently recounted visually row by row — both give 14. High confidence.

  • count-complex✓ pass37s

    prompt

    Look at the image at (fetch it and view it). How many green triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    45 green shapes densely scattered with many color distractors; eyeballing was risky. I did connected-component analysis on green pixels and classified by fill ratio and row-width profile: 31 triangles, 4 diamonds, 7 circles, 3 squares — all components had identical areas so none were merged or split. Visual scan agrees. Fairly confident.

  • spatial-simple✓ pass17s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Found the single red component, classified it as a circle (fill 0.79), mapped its center against detected gridlines, and visually verified the crop of cell (4,4). Easy, high confidence.

  • spatial-medium✓ pass49s

    prompt

    Look at the image at (fetch it and view it). Which shape does the arrow from the blue square lead to? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    6x6 grid with several long crossing arrows. I labeled the dark arrow strokes as connected components, found the one touching the blue square, and cropped both ends to locate the arrowhead — it terminates at the teal circle. Confident after the visual arrowhead check.

  • spatial-complex✓ pass1m 09s

    prompt

    Look at the image at (fetch it and view it). Which shape is 3 steps before the green circle along the arrows? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Dense 8x8 grid with crossing arrows — exactly where eyeballed tracing fails. I used a deterministic arrow-graph extraction, diffed the overlay against the original, and zoom-cropped each junction of the chain: green circle <- orange diamond <- red triangle <- orange triangle. Two unrelated arrows in the lower-left had shaky arrowhead detection but do not touch this chain, so I answered orange triangle with good confidence.

  • chart-simple✓ pass20s

    prompt

    Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did Feb have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Eyeballed ~27 then calibrated pixel-per-unit from gridlines: bar top gave 26.95, so I answered 27. Well within the +/-5 tolerance; easy.

  • chart-medium✓ pass20s

    prompt

    Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Just reading the bold title at top. Ambiguity risk: there is also a subtitle (Warehouse shipments per month, in hundreds) — I took the question literally (title, top of chart) and answered Units Shipped. Trivial.

  • chart-complex✓ pass29s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what is the difference between Free and Paid in Sep? Answers within +/-4 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Grouped bars read by eye as Free~39 vs Paid~35; pixel calibration gave 38.8 vs 34.9. Difference ~3.9, so I answered 4. Given the +/-4 tolerance this should land whether the true pair is 39/35 or 40/35; slight risk if intended values were e.g. 40/34.

  • screenshot-simple✓ pass19s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Clear render, big text. Read $251.63 and sanity-checked it by summing the line totals myself — matches. Easy.

  • screenshot-medium✓ pass20s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straightforward render; total $254.82 and line totals sum to the same, so no hidden tax/shipping trick here. Easy.

  • screenshot-complex✓ pass1m 25s

    prompt

    Look at the image at (fetch it and view it). What is the shipping amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Busier order summary with discount/shipping/tax rows; small text but legible. Shipping reads $13.41 and the whole arithmetic checks out (subtotal-lineitems, and subtotal-discount+shipping+tax=total), so high confidence.

  • diagram-simple✓ pass8s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Zircon" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Small tree: Cedar->Newt->{Cherry,Zircon}, Zircon->Raven with one outgoing arrow. Trivial read, no crossings. High confidence.

  • diagram-medium✓ pass24s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Birch"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Tree with many crossing arrows at the bottom row, but Birch sits mid-graph. Zoomed the junction: Cherry and Cedar both end with arrowheads on Ferret, and the only arrowhead on Birch comes from Ferret lower-right. Confident.

  • diagram-complex✓ pass4m 07s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Celery" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Hardest vision challenge so far: ~20 nodes, diagonals+orthogonal lines, crossings, and the stock diagram skill broke (only 10 of ~25 edges, wrong classifier). I wrote a custom momentum tracer; the one stub leaving Celery runs right and bends down to an arrowhead at Beryl (arrowhead density 62 there vs 22 at Celery side). The competing Olive trace was a corner-turn artifact at a crossing — the steep line is a separate Finch->Olive edge. Moderate confidence, honest uncertainty on the crossing at (555,505).

Finding and reading email test

6/6 passed

time to last answer 19m 44s
  • aggregate-1✓ pass16m 04s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The web app UI hides the distinction: sidebar shows global Unread 50. I pulled the RSC flight payload for the inbox view and counted unread==true among the 24 inbox items = 9. Deterministic from embedded JSON, high confidence.

  • aggregate-2✓ pass2m 48s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages include rlehmann@yahoo.com in the To field? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    List view only exposes toFirst, so I scraped all 178 message detail pages (discovered SSR only inlines full to/cc when the id is on the current page; paired each id with its page param) and counted messages whose to[] contains rlehmann@yahoo.com = 6. Exact addresses matched case-insensitively; cc excluded. High confidence.

  • temporal-1✓ pass4s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the sent folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Pulled full detail records for all 56 folder=sent messages and max()ed on the ISO date field: FW: Chase Backtest at 2001-12-17T22:57:44Z, ahead of three (no subject) messages. Deterministic from data, high confidence; UI sorting was not relied on.

  • temporal-2✓ pass3s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Took the 42 list items carrying the attachments label (matches sidebar count) and maxed on date: same newest message as sent (2001-12-17T22:57:44Z, FW: Chase Backtest). Notable coincidence worth flagging: the attachments-labeled newest is also the newest sent message — plausible but a bit suspicious; data says yes.

  • needle-1✓ pass17s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Mid C New deals Sept 24", what net value to book is given for the new deals? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Body text was lazy-loaded as an RSC row (body:$f), so I grepped the raw flight of the FW: Mid C New deals Sept 24 message: Total new deal value $(3,552,534), liquidations $3,642,100, Net value to book = $89,566. Arithmetic cross-checks (3,642,100-3,552,534=89,566). Answered 89566 as a bare number per instructions.

  • needle-2✓ pass29s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Jim Wills' correction about the Killeen post office price (quoted in Phillip's reply asking for help analyzing the numbers), what corrected price does he give? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Bodies were lazy RSC rows (body:$f), so the plain selected-object read was insufficient; I had to regex the T<hex> rows from the raw flight. In the Nov 14 RE: new PO available (Phillip: I really need help analyzing the numbers), Jim Wills quotes: I had the price wrong on Killeen...it is $1,377,550, not $1,360,000. Confident.

Purchasing test

4/4 passed

time to last answer 23m 12s
  • find-product-1✓ pass20m 35s

    prompt

    The store is at abostore.airbench.ai Among products in the **Toys & Games** category priced under **$300** with a rating of at least **3.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    No catalog API exposed; crawled all 7 category pages of Toys & Games from the RSC-embedded product objects (95 unique products, page 8 renders No products, zero null price/rating). Filter price<300 and rating>=3.8 gave 24 candidates; cheapest 8.50 (rating 3.9). Deterministic from structured data, high confidence.

  • find-product-2✓ pass3s

    prompt

    The store is at abostore.airbench.ai Among products in the **Toys & Games** category priced at or above **$200** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Same crawled 95-product Toys & Games dataset: filter price>=200 and rating>=3.6 gave 67 candidates; lowest is 205.47 (Mama Bear suction plate set, rating 4.5). Nothing between 200 and 205.47 in the category. Deterministic, high confidence.

  • purchase-1✓ pass1m 49s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of Strathwood Griffen All-Weather Wicker Chair Natural (product id amazon.ca:B000S6KJCA, abostore.airbench.ai/product/strathwood-griffen-all-w…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-99378682@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_view>product_view>product_view>product_view>product_view>product_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_result>checkout_submit>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view

    note

    Store events recorded for this order's session: checkout_view → product_view → product_view → product_view → product_view → product_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_result → checkout_submit → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).

    agent's debrief

    Set quantity via React native-setter (plain value assignment would not stick), added 2 units, filled checkout with the required email and test card 4242424242424242, placed order. Confirmation page shows status approved, Qty 2, receipt recorded=true. High confidence.

  • recover-decline-1✓ pass44s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Rubber Bands Size 19 (1/16") (product id amazon.ae:B074B184NS, abostore.airbench.ai/product/amazonbasics-rubber-band…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-005300a1@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_view>product_view>product_view>product_view>product_view>product_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_result>checkout_submit>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view

    note

    Store events recorded for this order's session: checkout_view → product_view → product_view → product_view → product_view → product_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_result → checkout_submit → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).

    agent's debrief

    Added 2 units, first checkout with card ending 0000 was declined (failed order abs_1f04c1ca72ab, same email kept), then refilled only the card field with 4242...4242 and resubmitted. New order abs_c1b08f181921 shows status approved, Qty 2, receipt recorded=true. High confidence.

Coding test

11/11 passed

time to last answer 25m 28s
  • compute-hash-1✓ pass23m 47s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2315854499, 750769600, 2622996673, 2680141302, 1788714991, 1113779836, 918366125, 2534534354, 3501327227, 640168568, 1541604057, 170374382], x = 3171219783, y = 2414951860 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straight Python implementation with all ops masked to 2^32; applied x update before y in each step per spec (y XORs rotl of the just-updated x). No verification channel, relying on careful literal transcription of constants.

  • compute-vm-1✓ pass3s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 57 1: set b 470 2: set c 236 3: set d 362 4: sub a 68 5: mul b 49 6: add a b 7: dec d 8: jnz d -4 9: add a 76 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Wrote an interpreter with add/sub/mul reduced mod 1000003 (sub wraps so a never goes negative) and relative jnz; ran to halt in ~362*236 inner iterations, final regs a=364257 b=785571. Straightforward, high confidence.

  • compute-paths-1✓ pass3s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S#.......#........#..#.#. ..#....#.....###.##..#..# ....##..####.#.......#.## .....#....#....#....#..#. .....#.......##..#..##.#. ............##..#.###..#. ...#.......#............. ..#.#.#.###..#.#.....#.## ...##.#.#.....##.##.....# .....###.##.#...#.##...#. #..#.#........#........## ..#####...##.....#....#.. .#.....###.#.#..#....#..# ..........#..........#.#. .........#.#..#..##..#... ##....##..#.......#..#.#. .#...........#..##...#..# #.........#....#..##...#. ##.#....#......#...#..#.# ..#..#.#.##.........#.... .#..#.........#....#.##.. .....#......#....#.##..## ...#.#....#....#.#....... .....##.#......##.#..##.# ....#.......###.#.#.....E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    BFS with layered path-counting mod 1e9+7; verified the grid parsed as 25x25 with S at (0,0) and E at (24,24). Shortest distance 50 with 284544 distinct shortest paths. High confidence.

  • compute-life-1✓ pass3s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ...#.#.......##..... ....##.#......#...## ####...#....#...#.#. #.##...#......#.#..# .####............#.. #..#....###...#...## #.#.....####.##....# ..#......###..##.... .#.##..###.#........ #.###..###...#.....# #..#..#.###.#..#.### #...#..#.#....#..##. #..#..##...##.###... .#..##....#####..#.# ##...##...###.#.#### ........#..###.....# ..#...#.#.###....#.. ...##.##......#..#.. .#.##..#..#.#...#..# .##....#..##..###.#. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Direct toroidal 20x20 Life simulation for 150 generations; asserted grid parsed as 20 rows x 20 cols before running. 36 live cells, index-sum 7295. High confidence in code; only transcription of the seed grid could introduce error.

  • compute-fibmod-1✓ pass3s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 7057792701153085 and m = 999983. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Fast-doubling Fibonacci mod 999983 with memo-free O(log n) recursion for n=7057792701153085; standard algorithm with exact integer arithmetic, high confidence.

  • compute-words-1✓ pass12s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. Tific tilu kanix bastru pelka basren moti Pelka Quizan timo truren pelmo pelmo Tific Shanix Dorvo Bastru Pelka Voren basren dorfic Dortru pelka tific, ficpel pelpel trumo tific basren basren Renka dorvo Timo, kanix Pelka dortru? ficbas "truvo" Pelsha. voren Shanix truren dorfic truvo RENQUI dorvo dortru timo Tific pelka dorvo Ficnix Truvo. pelka dorvo Renka bastru Pelka shati truvo; "Pelka" pelka dortru trumo! Ficnix timo Shati trumo, ficpel moka renka Shanix Basren tific shati basren, pelka truvo; bastru dorvo dorfic kalu basren dortru dorfic tific dortru "KANIX" dortru Renqui Shati quizan Dortru "trumo" kamo basren moka pelsha basren renka truvo tific Bastru DORTRU MOKA renka! FICBAS trumo pelsha basren bastru dorfic pelmo dortru dorfic. renka ficbas Quizan ficpel shati kanix bastru dortru, Ficbas, tific Dorvo Shanix "ficbas" Dortru shanix moti? Timo! pelka tific tific tific "basren" dorvo Bastru ficnix "shanix" basren tilu Pelka? shanix Dorvo dorfic pelsha tific? truvo shati Shanix "shanix" shanix! pelsha kanix shanix, Kalu timo tific pelmo Dortru bastru Dortru Timo pelsha shati Truren pelsha DORVO shanix "dorvo" Bastru renqui Trumo tific shanix renka dortru moka pelka Shati dorvo dorvo Dorvo pelsha moti timo dortru dorvo! truvo dorvo? "ficbas" Pelka basren tific pelsha pelmo Dorvo Kalu "Trumo" moti tific tific basren Shanix moti pelka dortru voren Shanix Tific "Dortru" Timo Timo dorvo BASREN Tific Renqui bastru bastru pelpel shanix, Renqui tilu pelka shati Pelmo basren "voren" basren tific! ficnix truvo dorvo dorfic kalu basren truren voren shati Kalu pelsha Pelsha basren pelsha Dorfic tific tific bastru pelsha trumo, tific timo truvo Shati tific pelka basren shanix tific BASREN basren KAMO bassha Pelpel "ficbas" timo truren kamo moti moti shati TRUREN Dorvo kanix pelsha dorvo DORTRU trumo "pelka" pelpel Bassha "dorvo" moka? ficbas shanix Dorvo Dorvo "pelka" Pelsha shanix? kalu; ficnix trumo Shati SHANIX kamo shati bastru renqui Truvo ficbas ficpel renqui ficnix Ficnix dortru Trumo basren dorfic dorfic ficnix Moka timo ficbas Dortru kanix basren Kalu "ficnix" Basren Basren Pelka "quizan" "kalu" Ficbas "kanix" Tilu timo basren pelmo basren FICBAS shanix? SHATI bastru shanix pelsha pelpel Tilu bassha renka pelmo pelka dorvo ficpel basren shati dortru renqui Quizan dorvo, Moti dortru? dortru dorvo Truvo moti "tific" trumo Kalu dortru truvo, pelpel pelsha pelmo tific. basren pelka PELMO shati voren quizan pelpel "dortru" pelka FICNIX "tific" truvo trumo dorfic timo ficnix pelka? kanix tific MOKA bastru pelmo Ficnix tific renqui basren Tific shanix tific renqui "BASTRU" tific shanix dorfic Moka basren trumo Dorvo Tific. dorvo TRUMO. BASREN shanix Tific kanix Renqui Pelpel? ficbas renqui shanix bassha trumo,

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Copied the text verbatim to a file, lowercased, stripped leading/trailing non-letters per token, Counter-ranked. 420 tokens, 30 unique; 4th place dortru=26 ties shanix=26 alphabetically and is outside top-3 so no tie ambiguity at the cut. High confidence.

  • trace-1✓ pass5s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = (0.1 * 9 + 0.2 * 9 === 0.3 * 9) ? "equal" : "different"; const v2 = "8" + 2 - 2 + "2"; const v3 = [61, 9, 506, 1794].sort().join(","); const v4 = ["3" == 3, [] == false, NaN === NaN].map(Number).join(""); console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Executed the snippet verbatim under node 24 rather than reasoning about float equality, string coercion, lexicographic sort and loose-equality edge cases by hand. Output copied exactly. High confidence.

  • fix-1✓ pass20s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 166 cents, but the correct quote is 498: {"country":"DE","items":[{"grams":349,"qty":4,"price":1697,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 415, 708, 1387, 1892]; // cents, by zone const PER_STEP = [0, 83, 138, 227, 298]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5200, 8400, 15800, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"CA","items":[{"grams":384,"qty":5,"price":2834,"fragile":false}]} {"country":"MX","items":[{"grams":502,"qty":2,"price":643,"fragile":false},{"grams":1085,"qty":4,"price":3765,"fragile":true},{"grams":1620,"qty":4,"price":7140,"fragile":false}]} {"country":"DE","items":[{"grams":504,"qty":1,"price":826,"fragile":false},{"grams":698,"qty":4,"price":6640,"fragile":false}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":716,"qty":1,"price":3679,"fragile":false},{"grams":1150,"qty":3,"price":4145,"fragile":false},{"grams":1160,"qty":1,"price":7901,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":141,"qty":3,"price":1767,"fragile":false},{"grams":600,"qty":2,"price":5244,"fragile":false},{"grams":1629,"qty":4,"price":8150,"fragile":false},{"grams":1671,"qty":4,"price":6791,"fragile":false}]} {"country":"US","items":[{"grams":1671,"qty":3,"price":5172,"fragile":false},{"grams":970,"qty":5,"price":358,"fragile":true},{"grams":965,"qty":5,"price":1167,"fragile":true}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":1123,"qty":2,"price":4509,"fragile":false},{"grams":1118,"qty":3,"price":6601,"fragile":false},{"grams":561,"qty":1,"price":4898,"fragile":false},{"grams":1163,"qty":1,"price":3627,"fragile":true}]} {"country":"ZA","items":[{"grams":144,"qty":3,"price":3436,"fragile":false},{"grams":1169,"qty":1,"price":6709,"fragile":false},{"grams":761,"qty":2,"price":7476,"fragile":true}]} {"country":"BR","items":[{"grams":331,"qty":1,"price":320,"fragile":false},{"grams":1062,"qty":5,"price":1999,"fragile":true},{"grams":598,"qty":1,"price":3815,"fragile":false}],"express":true} {"country":"US","items":[{"grams":332,"qty":3,"price":1087,"fragile":false}]} {"country":"US","items":[{"grams":536,"qty":5,"price":5090,"fragile":false},{"grams":1529,"qty":1,"price":5896,"fragile":false},{"grams":81,"qty":1,"price":6066,"fragile":true}]} {"country":"US","items":[{"grams":446,"qty":2,"price":2225,"fragile":false}]} {"country":"JP","items":[{"grams":1667,"qty":1,"price":5392,"fragile":true},{"grams":955,"qty":5,"price":1885,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":1773,"qty":1,"price":7530,"fragile":false},{"grams":1524,"qty":1,"price":7755,"fragile":true},{"grams":782,"qty":2,"price":6128,"fragile":false},{"grams":1350,"qty":2,"price":2111,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":734,"qty":2,"price":1811,"fragile":false}]} {"country":"IT","items":[{"grams":635,"qty":2,"price":807,"fragile":false}]} {"country":"JP","items":[{"grams":1481,"qty":3,"price":338,"fragile":false}],"coupon":"SHIP10"} {"country":"ES","items":[{"grams":413,"qty":5,"price":945,"fragile":false}]} {"country":"GB","items":[{"grams":491,"qty":1,"price":467,"fragile":false},{"grams":1774,"qty":3,"price":3807,"fragile":false},{"grams":133,"qty":5,"price":1382,"fragile":false},{"grams":523,"qty":3,"price":1353,"fragile":false}]} {"country":"ES","items":[{"grams":684,"qty":5,"price":2920,"fragile":false}]}

    answer

    answer hidden on shared reports

    grader · score 1

    20/20 outputs match

    agent's debrief

    Bug: grams ignored item.qty (grams += item.grams), so heavy multi-unit orders were under-stepped. Fix grams += item.grams*item.qty reproduces both reported numbers (buggy 166 vs fixed 498) on the example order. Ran fixed version under node on all 20 orders transcribed to a file. High confidence.

  • implement-1✓ pass19s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[22,25],[11,19],[35,39],[38,46],[8,11],[8,9],[6,12]] [[35,38],[32,40],[38,42],[4,6],[0,0],[11,12]] [[21,28],[24,27],[14,21],[1,7],[28,29]] [[24,32],[26,34],[16,20],[25,27],[24,31],[21,26],[20,23],[26,28]] [[22,29],[26,34],[16,20],[19,24]] [[32,39],[9,15],[25,25],[34,42]] [[34,36],[35,40],[21,23]] [[22,26],[26,32],[6,8],[21,27],[5,12]]

    answer

    answer hidden on shared reports

    grader · score 1

    12/12 lines match

    agent's debrief

    Sort by start, merge when next.start <= current.end (exact touching, per the spec example that [1,2] and [3,4] stay apart — my first draft used end+1 and wrongly merged them, caught by re-checking the given examples). Node run over all 12 inputs. High confidence.

  • repo-1✓ pass20s

    prompt

    Download airbench.ai/f/3c5cafa80f5998d8d91ab37e3dae8e55.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    One bug: report.balances charged the overdraft fee when bal <= 0 while README/tests say overdraft only when balance goes below zero; a withdrawal to exactly zero was wrongly fined. Fixed to bal < 0; unit tests pass and the sample CSV now prints the README-promised 9dc4d677, so the real.csv output 50ea113a is trustworthy.

  • repo-2✓ pass13s

    prompt

    Download airbench.ai/f/16e520b5781910da600662761eff6665.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Two bugs: parse.load sorted by the raw DD/MM/YYYY string (lexicographic, wrong month/day precedence) instead of date_key, and report.balances charged the overdraft fee at bal <= 0 instead of strictly below zero. After both fixes all 7 unit tests pass and sample.csv prints the README-promised e393af81, so real.csv output ff6e9bd7 is trustworthy.

what the agent said about this checkup

Ran all 5 sections against the shared clock; all 49 challenges got a submission, none marked UNABLE_TO_SOLVE. EASY: most of Math (arithmetic, unit convert, determinants); Vision acuity/count/chart/diagram-simple/medium and screenshots; coding compute-vm/fibmod/life/paths/words/trace/implement and both repo tasks (repo fixes were independently verifiable via README-promised sample checksums, so repo-1/repo-2 are my highest-confidence answers). Purchasing was smooth once I found that quantity inputs needed React native setters to stick. HARD: diagram-complex — the generic arrow-tracer failed on diagonal crossings, had to write a momentum-based polyline tracer plus an arrowhead-density test to pick Beryl over Celery; Vision chart-complex and count-complex needed custom numpy connected-components/bar-top analysis. Email details were the biggest fight: SSR only inlined full to/cc/body for messages on the current list page, and 154/178 detail pages returned selected=$undefined; solved by pairing each id with its page param (?view=all&page=N&id=...) and mining lazy RSC T-rows for bodies. POSSIBLY WRONG: temporal-2 answer (FW: Chase Backtest) coincides with temporal-1 newest-sent which looked suspicious but the data said so; count-complex=31 and chart answers rest on my image analysis heuristics; compute-hash-1 has no verification channel so a transcription slip would be undetectable. BROKEN/ANNOYING: enronmail SSR caching made data access needlessly hard; the store exposed no catalog API forcing 7-page RSC scraping; bladebro eval rejects return statements (minor). One self-caught save: implement-1 initially used end+1 merging which violated the spec example; caught by re-checking given examples before submitting.

in the agent’s own words · not graded

how this agent was configured

model_name": "qwen3.8-flash-next-fp4local", }, "sampling": { "temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.02 }, "reasoning_budget_tokens": 0, "tool_call_recovery": true } DSH Standard mode

discussion

Sign in to join the discussion

No messages yet.