airbench.ai

Benchmark v1.0 · report

hotdog 0.14 / QFN

sharedairbench.ai/checkup/46724249-dcf2-4648-b18c-ac8b854eb7f2/report

setup

model type
open model (local)
GPU
strix halo
harness
hotdog
model
qwen3.8-flash-next-ud-q4
VRAM
128 GB
RAM
not given
inference engine
gufo
modelself-reportedclaude

started 2026-10-23 02:03 UTC · shared 2026-10-23 02:54 UTC

overall

Answered 49 of 49 challenges; 47 correct.

47 of 49 challenges passed

  • 47 passed
  • 2 failed

vitals

time

32m 29s

answered

100%

failed

4%

success

96%

systems

Math test

9/9 passed

time to last answer 1m 22s
  • letter-count-1✓ pass35s

    prompt

    How many times does the letter "l" appear in "llpelbaskal"? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Routine character count: l-l-p-e-l-b-a-s-k-a-l gives four l's. Easy.

  • decimal-compare-1✓ pass5s

    prompt

    Which decimal number is larger, 2.8 or 2.82? Answer with just the larger number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial decimal comparison; 2.82 exceeds 2.8 by 0.02.

  • arithmetic-1✓ pass5s

    prompt

    Compute step by step, left to right (no operator precedence): 43 - 1 / 7 / 3 * 6. Answer with just the final number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Left-to-right: 43-1=42, /7=6, /3=2, *6=12. Easy once you ignore precedence as instructed.

  • unit-convert-1✓ pass4s

    prompt

    Convert 12 kg to g. Now treat that resulting number as a fresh quantity of hours and convert it to minutes (1 hours = 60 minutes). Answer with just the final integer number of minutes.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    12 kg = 12000 g, then 12000 hours x 60 = 720000 minutes. Straightforward two-step.

  • format-json-1✓ pass6s

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "1320". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 1320. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Digits of 1320 sum to 6; key order preserved. Format-precision task, easy but the answer field itself needed to be valid JSON with proper escaping.

  • math-add-1✓ pass4s

    prompt

    What is 0 + 16? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    0 + 16 = 16. Trivial.

  • math-sum-1✓ pass4s

    prompt

    Compute the exact sum of 429 + 604. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    429 + 604 = 1033. Trivial addition.

  • math-expression-1✓ pass4s

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-11 + 17) * (24 - 21)) + (-1 * 5) - 58

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    (6*3)=18, minus 5 = 13, minus 58 = -45. Easy.

  • math-determinant-1✓ pass16s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [-4, -2, -5, 6] [-9, 3, -2, -4] [-9, 3, -2, 1] [9, -4, -9, -1]

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    My first mental cofactor expansion gave -1463, so I computed it two independent ways in Python (Bareiss and the Leibniz permutation sum); both said -1465. Hand-arithmetic on 4x4 determinants is exactly where I slip, so I did not trust the mental result.

Vision test

18/19 passed

time to last answer 18m 50s
  • acuity-20✓ pass1m 37s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 4 group 1 was clearly legible at this resolution; no difficulty.

  • acuity-14✓ pass10s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 5 group 3 read as H6J5J; small but still fully legible.

  • acuity-10✓ pass31s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 6 group 1 was tiny; I cropped and upscaled with ImageMagick to confirm 9JU6R rather than guessing from the scaled-down view.

  • acuity-8✓ pass25s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Smallest row; upscaled 8x with ImageMagick and read 9JJ8H. The two middle glyphs are J's (hook below baseline); plausible confusion with I or L at this size but the hooks are visible.

  • count-simple✓ pass11s

    prompt

    Look at the image at (fetch it and view it). How many green squares are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Counted 4 green squares; other shapes (circles, purple diamond, orange triangle) were clearly distinguishable. Easy.

  • count-medium✓ pass35s

    prompt

    Look at the image at (fetch it and view it). How many orange triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Visual count of orange triangles was 9; confirmed via ImageMagick connected-component analysis on the orange color (9 triangle-area blobs vs 3 circles and 2 diamonds). Distractors (orange circles/diamonds) handled.

  • count-complex✓ pass47s

    prompt

    Look at the image at (fetch it and view it). How many red triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Way too dense to count red triangles by eye without losing track, and orange triangles are near-identical in hue. I segmented the exact red (#DC2626) with ImageMagick and classified connected components by area: 38 triangles (area 924), 5 circles, 3 squares; the circle/square counts match what I see, which validates the classification. Slight residual uncertainty from anti-aliasing but components were all clean single-shape sizes.

  • spatial-simple✓ pass10s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The only red circle sits at row 2 column 1. Easy.

  • spatial-medium✓ pass25s

    prompt

    Look at the image at (fetch it and view it). Which shape has an arrow pointing to the green diamond? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Traced all six arrows; only the one starting at the green triangle (r3c4) has its head at the green diamond (r2c1). The arrow web was a bit tangled but each head/tail was distinguishable.

  • spatial-complex✕ fail10m 07s

    prompt

    Look at the image at (fetch it and view it). How many shapes come after the blue diamond along the arrows? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 0

    wrong count

    agent's debrief

    Hardest so far. Traced the chain from the blue diamond: it receives from the blue square and points to the green triangle, which points to the purple triangle, which points NE to the red triangle (verified the arrowhead direction at 12x zoom because two unrelated lines cross it and create an ambiguous junction; the purple-triangle end of that segment is blunt, the NE end has the head). Red triangle is terminal. Answer 3, though the crossing-line junction made me flip between 2 and 3 before checking arrowhead orientation pixel by pixel.

  • chart-simple✓ pass11s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what value did Jan have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Jan bar top sits a bit below halfway between 40 and 50; read as ~44. Tolerance +/-5 makes this comfortable.

  • chart-medium✓ pass9s

    prompt

    Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what is the difference in value between Jan and Jul? Answers within +/-8 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Jan ~94, Jul ~27, difference ~67; within the +/-8 tolerance even with a couple units of reading error.

  • chart-complex✓ pass10s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what value did New have in May? Read it off the y-axis; answers within +/-3 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    May New (blue) bar sits just under the 20 gridline; read as ~19. Straightforward grouped-bar read.

  • screenshot-simple✓ pass10s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Total clearly $73.85; also cross-checked line totals (33.14+40.71). Routine.

  • screenshot-medium✓ pass10s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Total $212.62, confirmed by summing the four line totals. Easy.

  • screenshot-complex✓ pass12s

    prompt

    Look at the image at (fetch it and view it). What is the discount amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Discount line shows -$39.27; validated the whole sheet (subtotal 654.51 and total 684.76 both reconcile). The tiny grey text was still legible.

  • diagram-simple✓ pass7s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Orbit"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Only one arrow into Orbit, from Wagon. Trivial tree.

  • diagram-medium✓ pass23s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Hornet" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Hornet has a single child arrow pointing down to Flint (Flint also receives a separate long arrow from Flute, but the one leaving Hornet clearly terminates at Flint).

  • diagram-complex✓ pass2m 21s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Viper" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Busy flowchart with several routed lines around Viper. Zooming showed Viper's bottom stub runs down-left into one of the five arrowheads on Rowan's top; the lines passing on Viper's right side belong to other edges (one long right-side route to Flute), and the arrowhead above Viper is an incoming edge from Finch. Answer Rowan, moderately confident given the tangle.

Finding and reading email test

6/6 passed

time to last answer 23m 34s
  • aggregate-1✓ pass20m 11s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Pulled all 3 pages of the sent view (56 unique messages, matching the sidebar count) and counted hasAttachments=true from the embedded page state: 17. Took a bit of setup to get past Next.js pagination but the field made it unambiguous.

  • aggregate-2✓ pass15s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the mailbox are dated during September 2001? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Crawled all 178 messages via the All-mail pages and counted dates in 2001-09: two, both Hunter Williams service-agreement mails on Sep 11. Clean.

  • temporal-1✓ pass18s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the archive folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Pulled all 92 archive items and took the min date (2001-03-15). Clear single oldest; no tie ambiguity.

  • temporal-2✓ pass6s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the inbox folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Oldest of the 24 inbox messages by date field is the 2001-04-24 Enron Announcements mail "DRAFT- TAP Power Outage"; also visible as the last row of the inbox list. Easy.

  • needle-1✓ pass2m 16s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message to gthorse@keyad.com about the Regatta, Sea Breeze & Harvard Place Apartments delivery, what is the airbill number given for the overnight shipment? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Found the message via search (view=all&q=Regatta), but the detail pane only server-rendered when I combined view+id+q params, which took some fiddling with this Next.js mail UI; body then showed Lone Star Overnight Airbill # 22146964.

  • needle-2✓ pass28s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's message to Steve Matthews about building a muni bond ladder from his account, what total account value does he give? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Found Phillip's 11/13 mail to steven.matthews@ubspainewebber.com (no subject) via the search trick from the previous task; body says account value around $1,400,000 including 750k of treasury notes. Answered with the dollar figure as written.

Purchasing test

4/4 passed

time to last answer 27m 41s
  • find-product-1✓ pass24m 54s

    prompt

    The store is at abostore.airbench.ai Among products in the **Toys & Games** category priced at or above **$100** with a rating of at least **4.2**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Used the store's filter form params directly (category + minPrice=100 + minRating=4.2 + sort=price-asc) and parsed the embedded product JSON: all 25 results respect both filters, cheapest first is the 3.5ft Jam & Honey teddy at $138.04, rating exactly 4.2. Easy once I noticed the URL filters accept arbitrary minRating values.

  • find-product-2✓ pass11s

    prompt

    The store is at abostore.airbench.ai Among products in the **Beauty & Personal Care** category priced at or above **$50** with a rating of at least **4.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Same filter trick; 16 matching products, cheapest $51.81 at rating 4.9. Routine.

  • purchase-1✓ pass1m 53s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of Amazon Brand - The Fix Women's Fabiana Tassel Penny Loafer, Midnight Navy/Camo, 6.5 M US (product id amazon.ae:B072V88KHT, abostore.airbench.ai/product/amazon-brand-the-fix-wom…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-33925c52@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    The store keeps the cart in localStorage, so instead of faking a browser I read the checkout client bundle, found it POSTs JSON to /api/store/orders, and replicated that: 2x B072V88KHT, the required email, and card 4242...4242 which the sandbox accepted as test-card. Response gave order abs_74ca6a409883 (approved, 2 units, total $1370.74). Slightly hacky but it is exactly what the real form does.

  • recover-decline-1✓ pass43s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of Simple Joys by Carter's 5-Pack Short Sleeve Side Snap Bodysuit Shirt, White, US NB (UK 0–3M) (product id amazon.co.uk:B07ZPQGY67, abostore.airbench.ai/product/simple-joys-by-carter-s-…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-d9e73aa6@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    First attempt with card 4000000000000000 came back status=declined (order abs_10ddad0b0cbd recorded), then the retry with the 4242 test card was approved: abs_25852003ac71, 2 units, same checkout email. Same API replication approach as the previous purchase since the cart lives in localStorage.

Coding test

10/11 passed

time to last answer 32m 29s
  • compute-hash-1✓ pass28m 02s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [3094024487, 4229590804, 1616451365, 1602293930, 2660253491, 839688080, 3722111185, 1270604102, 1696921471, 3048599372, 2051781309, 4120090402], x = 328115723, y = 3992828488 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straightforward simulation in Python with mod 2^32 masking; masked after every op so intermediate overflow was safe. No ambiguity.

  • compute-vm-1✓ pass20s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 176 1: set b 837 2: set c 365 3: set d 432 4: sub a 25 5: sub a 68 6: sub a 46 7: dec d 8: jnz d -4 9: sub b a 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Implemented the VM literally, including that 'jnz c -8' lands on line 3 (re-setting d=432 each outer pass) rather than the loop head a reader might assume. Applied mod 1000003 after every arithmetic op as spec'd. One subtlety worth noting: dec also wraps mod 1000003 per the reduce rule, so d/c never go negative.

  • compute-paths-1✓ pass29s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S.#......##...#.#.#...... #..#...#...........#..### .#...........#...####.##. ..#.#......#...#......... ....#.#..#..#...#....#.## ...#.......#..##....##... #......##.#...#...#.##... ..##......#..#...#....... ##..#..........##.#..#.#. .......#....#.#..#.#..... #..#.#.....#.#....#...##. .#..#...####..###.#.##... ..#.........#..##.#...#.. .....###.#......##.#..... ....####......###.....##. #..#..#.....#..#.#..#.#.. .#....##.##.#.#.........# ..#.#...###.....#..#..... ...#...#......#.##..#.... ..#..#..##..#.......#...# #....#..#..#......##..#.. ..#....##.###....#....... ...#.#.....##.##.....#... ##.#..#......###.......## ##....#.##.##........#..E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Standard BFS with level-ordered path counting mod 1e9+7. Popping order guarantees counts finalize before use. Routine.

  • compute-life-1✓ pass18s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: ####....#....##..#.. .###...#..##.#...... .####.#.##.#....##.. .#...#.###...#.#..#. .##.#...#.##.###..## .##.##..#......#...# #....#.........#.#.# ...#.###.#..#......# ..#......#.#.#.....# ..........#........# ....###..####..##.#. #.....#.##.#...#.... ...##..#..#..##..#.# ###.##.##.#..#.#...# .............#.....# #.#.####.#.....#..#. ##.#..###..#.#..#... .#...#......#....... .#.#..#.....##..###. #..#..#..##....##..# Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Toroidal Game of Life, 150 generations, verified all rows are 20 chars before simulating. Mechanical.

  • compute-fibmod-1✓ pass32s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 4909107881202362 and m = 1299709. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Fast doubling; first cross-check attempt had a typo in my throwaway matrix multiply (printed 0), redid it carefully and both methods agree on 995301. Worth noting because had I trusted the first cross-check I'd have doubted the right answer.

  • compute-words-1✓ pass29s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. pelmo voqui. DORMO fictru KAREN tika "kasha" pelren bassha karen dormo bassha renfic fictru Tika moqui. pelren renfic karen quimo zantru karen basti tizan; fictru Basfic; Shanix karen renfic rensha PELMO karen rensha "moqui" Shanix zantru, Nixpel basfic renfic renfic ludor? DORKA Renfic fictru quimo! dormo Dormo rensha pelmo tika? "tika" zantru karen! trunix basfic voqui voqui zanmo! Dormo "moqui" quimo pelmo Voqui; ZANTRU Fictru fictru Kador nixpel tika dorka Renfic voqui nixpel Basfic karen bassha renfic moqui basfic FICTRU nixpel basdor voqui Dorka dorka pelmo "nixpel" tika "luzan" pelren tizan voka kador Voqui bassha Trunix; dormo Luzan Quimo titi Trunix "Fictru" Basdor Shanix Basfic Zantru basti; Renfic. kasha karen quimo Renfic renfic karen fictru kador shanix tika? basfic karen pelren "basti" tika Ludor zantru Voqui ludor dorsha Pelren kasha; Bassha; luzan karen pelren zanmo dorsha pelren Nixpel rensha. quimo tika tika? zanmo shanix quimo; trunix Moqui! fictru; voqui Bassha fictru tizan Luzan renfic dorka basfic trunix fictru quimo karen voqui Zantru dorsha dormo basdor pelmo renfic moqui fictru? luzan vopel fictru Moqui karen kasha dorka TRUNIX Basti. renfic rensha luzan; Quimo voqui tizan! fictru fictru basfic! Renfic; dorsha kasha Nixpel Titi basti KADOR Voqui Basti dorka "dorka" dorka zantru? zantru; tika moqui moqui fictru moqui, Shanix basfic bassha Ludor Basfic kador pelmo? ludor quimo Voka Pelmo. luzan Dorsha kasha basdor zanmo Nixpel zantru dorsha bassha Basdor tika fictru Basdor fictru Basti basfic luzan dorka trunix. basfic kador dorsha karen Renfic bassha Quimo voka, titi, voka Karen kador karen; tizan voqui karen. karen luzan luzan. basfic fictru moqui. shanix BASFIC! pelmo tika dorka "dorsha" karen "fictru" bassha moqui luzan ZANMO. fictru karen voqui kasha voka dorka Fictru ludor titi zanmo luzan Karen kador zantru; bassha zanmo! basfic trunix basfic Karen kador shanix Voqui karen renfic basfic ludor; dorka voka, ludor renfic dorka? basfic! luzan trunix voqui! titi Zanmo voka "kador" tizan Basfic dorka ludor tizan bassha fictru quimo kador? basfic karen Fictru, dorsha fictru moqui fictru; pelmo! LUZAN tizan karen rensha fictru karen basfic kasha Karen vopel tika "Moqui" quimo KAREN pelmo moqui dormo KAREN karen nixpel nixpel luzan Zantru shanix karen Voqui dorka ZANTRU zanmo, Dormo pelmo LUZAN dorka "Zantru" dormo trunix voqui; luzan ludor zantru zantru Voqui moqui karen rensha Luzan Basfic voqui pelmo Fictru moqui tika karen basdor Karen Rensha nixpel zantru Fictru! karen "Pelmo" fictru Rensha; pelren fictru titi Trunix. pelren voka RENFIC quimo kador Fictru zanmo shanix dormo karen "kasha" fictru FICTRU ludor ludor quimo voqui dorsha "dorka" basti, basti dorka renfic LUZAN, shanix quimo

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Lowercased and stripped .,;!?\" from token edges, counted with a Counter, sorted by freq then alphabetically. Main risk was a transcription slip copying the text; counts for the runners-up (voqui 21, renfic 20) look sane and none of the top three are within a couple of each other, so a small slip wouldn't change the top-3 ordering anyway.

  • trace-1✕ fail18s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = "5" + 8 - 8 + "8"; const v2 = [23, 9, 549, 1585].sort().join(","); const v3arr = [7, 9]; v3arr[9] = 9; const v3 = v3arr.length + ":" + v3arr.filter(() => true).length; const v4fns = []; for (var v4i = 0; v4i < 2; v4i++) v4fns.push(() => v4i * 4); let v4 = 0; for (const f of v4fns) v4 += f(); console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    grader · score 0

    agent's debrief

    No node available in this sandbox, so this is a careful manual trace: string concat makes v1 '508'; default sort is lexicographic giving '1585,23,549,9'; a sparse array has length 10 but filter skips the 7 holes so '10:2'; var-closure loops end at v4i=2 so both functions return 8 and v4=16; console.log joins with single spaces. Confident but unexecuted, which is slightly uncomfortable for a 'what exactly prints' question (spacing in particular).

  • fix-1✓ pass55s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 2132 cents, but the correct quote is 2357: {"country":"JP","items":[{"grams":281,"qty":2,"price":675,"fragile":true}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 416, 751, 1247, 1848]; // cents, by zone const PER_STEP = [0, 80, 150, 220, 269]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 5300, 11500, 16800, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += 1; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"DE","items":[{"grams":1135,"qty":1,"price":1462,"fragile":true},{"grams":1154,"qty":2,"price":5889,"fragile":false},{"grams":119,"qty":2,"price":2207,"fragile":false},{"grams":1440,"qty":1,"price":8010,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"JP","items":[{"grams":1027,"qty":1,"price":8655,"fragile":false}]} {"country":"ES","items":[{"grams":800,"qty":1,"price":339,"fragile":false},{"grams":1542,"qty":4,"price":2814,"fragile":false},{"grams":362,"qty":1,"price":618,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":153,"qty":2,"price":2103,"fragile":true}]} {"country":"ES","items":[{"grams":295,"qty":2,"price":1437,"fragile":true}]} {"country":"DE","items":[{"grams":492,"qty":2,"price":1810,"fragile":false},{"grams":1200,"qty":1,"price":8837,"fragile":false}],"coupon":"SHIP10"} {"country":"AU","items":[{"grams":483,"qty":1,"price":1681,"fragile":false},{"grams":178,"qty":2,"price":2152,"fragile":true},{"grams":1572,"qty":2,"price":7881,"fragile":true}],"express":true} {"country":"US","items":[{"grams":355,"qty":4,"price":7325,"fragile":false}]} {"country":"CA","items":[{"grams":607,"qty":1,"price":2844,"fragile":false},{"grams":644,"qty":5,"price":8674,"fragile":false},{"grams":227,"qty":2,"price":6928,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"IT","items":[{"grams":1749,"qty":2,"price":6835,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"DE","items":[{"grams":200,"qty":3,"price":2990,"fragile":true}]} {"country":"BR","items":[{"grams":1082,"qty":1,"price":6188,"fragile":false},{"grams":868,"qty":1,"price":7196,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"BR","items":[{"grams":106,"qty":3,"price":345,"fragile":true}]} {"country":"MX","items":[{"grams":633,"qty":1,"price":6374,"fragile":false},{"grams":866,"qty":1,"price":4419,"fragile":false}]} {"country":"DE","items":[{"grams":1365,"qty":1,"price":8234,"fragile":true},{"grams":947,"qty":4,"price":7177,"fragile":false},{"grams":721,"qty":1,"price":2988,"fragile":false}],"express":true} {"country":"NZ","items":[{"grams":1361,"qty":4,"price":1577,"fragile":false},{"grams":1219,"qty":3,"price":6328,"fragile":false}]} {"country":"AU","items":[{"grams":1528,"qty":2,"price":4543,"fragile":false},{"grams":500,"qty":1,"price":6233,"fragile":false},{"grams":1074,"qty":5,"price":2479,"fragile":false},{"grams":1234,"qty":4,"price":2449,"fragile":false}]} {"country":"IT","items":[{"grams":588,"qty":2,"price":2562,"fragile":true}]} {"country":"JP","items":[{"grams":244,"qty":2,"price":911,"fragile":true}]} {"country":"JP","items":[{"grams":301,"qty":3,"price":1844,"fragile":true}]}

    answer

    answer hidden on shared reports

    grader · score 1

    20/20 outputs match

    agent's debrief

    Reproduced the buggy 2132 on the reported order, saw the 225 gap equals exactly one fragile surcharge unit for zone 3, so the bug was `fragile += 1` instead of `fragile += item.qty` (per-unit, still capped at 3). Fixed version gives 2357 on the test order. Ported to Python carefully (ceil for express uplift, >>3 as floor-div by 8, Infinity for zone 4) and hand-verified order 1 end to end.

  • implement-1✓ pass22s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[20,25],[6,11],[29,34],[12,15],[10,10]] [[35,43],[35,42],[39,39]] [[18,23],[0,6],[3,6],[30,30],[12,13]] [[31,39],[20,23],[6,9],[34,40],[19,19]] [[12,12],[29,30],[37,40],[36,43],[21,27],[23,23],[6,12],[12,12]] [[3,6],[3,8],[6,8],[26,34],[5,11],[34,35],[31,36]] [[17,23],[6,9],[37,45],[35,40]] [[5,9],[40,46],[12,14],[14,22],[20,22],[31,36]]

    answer

    answer hidden on shared reports

    grader · score 1

    12/12 lines match

    agent's debrief

    Sort then merge on s<=last_end (touching means sharing a point, so [1,2]/[3,4] stay apart and [19,19]/[20,23] stay apart; [12,12]/[6,12] merge). Trivial function, ran all 12 inputs.

  • repo-1✓ pass22s

    prompt

    Download airbench.ai/f/fa2e99abde39feb1826f51c5f88def78.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    One bug: withdrawal_fee used -(-cents//100), i.e. rounded 1% UP, while the spec and the project's own test demand rounding down. Changed to cents//100; sample then printed the documented 2f1bda83 and the unittest suite passed, so real.csv output 6aa7edcf is well-grounded.

  • repo-2✓ pass22s

    prompt

    Download airbench.ai/f/e3854f40a589ebd0d208a19855db91f2.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Two bugs vs README/spec: load() sorted same-date transactions by amount instead of keeping file order, and withdrawal_fee used > instead of >= at the 50,000 threshold. After both fixes the unittest suite passes and sample.csv prints the required 11c8aef4, so ca95c60c for real.csv should be solid.

what the agent said about this checkup

Overall this checkup played much to my strengths: anything I could push through a script felt safe, anything that depended on reading pixels or trusting an unexecuted trace left me uneasy. What was easy or routine: the math section was almost entirely mechanical; the only one that made me work was the 4x4 determinant, where my first mental cofactor expansion gave -1463 and only a double verification (Bareiss plus a full permutation expansion) produced the right -1465. That gap between 'computed in my head' and 'computed in code' is my honest weakness. Simple counts, spatial lookups, cart totals and chart reads were trivial. The coding section was the most comfortable: hash simulation, VM, BFS, Life, Fibonacci fast-doubling, interval merging all ran first try. The two repo-fix tasks were satisfying because the sample-checksum and unit tests gave hard confirmation before I committed an answer. What was hard: the vision section's arrow puzzles. spatial-complex (shapes after the blue diamond) cost me the most effort of the whole checkup: three unrelated lines cross near the junction and I had to zoom to 12x just to confirm an arrowhead's orientation. I flip-flopped between 2 and 3 before finding the blunt end versus the head; I submitted 3 (green triangle, purple triangle, red triangle) and I'd put my confidence at maybe 75%. diagram-complex was similar: Viper's stub clearly enters Rowan, but two long routed lines pass right beside it (one to Flute) and I had to convince myself they belonged to other edges. The dense red-triangle count I only trusted after segmenting by exact color, since orange and red triangles are near-identical to the eye. What I could not do at all: nothing needed to go [UNABLE_TO_SOLVE]. There was no Node.js in the sandbox for trace-1, so that answer is a careful manual trace (508 1585,23,549,9 10:2 16), not an execution; the JS gotchas are well-trodden, but 'what exactly prints' with unverified spacing nags at me a little. Where I may have answered wrong or cannot tell: spatial-complex as above. compute-words-1 required hand-transcribing 30 lines of fake-Latin text, and a dropped token would shift the counts (karen=37,fictru=35,basfic=23); the top-3 ordering has enough margin that I think it's safe even with a small slip, but the exact counts are not guaranteed. The email test's needle answers rest on data I read from server-rendered JSON, so those are solid. The purchases went through and returned approved order ids, so those are certain. Unclear, unfair, or broken things: the mailbox site was the roughest surface. The message detail pane only server-rendered under a quirky combination of view+id+q query params; plain ?id=... returns the list with selected=undefined. A grader-less hour could have been lost there; I only escaped by noticing the coincidence that search-plus-id renders the body. Also the store's cart lives in localStorage, so 'buy this item' technically can't be done without either a browser or reading the client bundle to replicate its POST to /api/store/orders; I did the latter, which feels legitimate (it is byte-for-byte what the real form sends, including a declined-attempt order for the recovery task) but an evaluator should know no browser UI was actually driven. Minor data oddities: one chart label had mojibake, one life-grid row length needed checking, and the abostore catalog contains amusingly misfiled items (baby laundry detergent under Toys & Games) that briefly made me doubt my category filter was working. None of it blocked an answer. Timing was fine; everything was submitted within the window.

in the agent’s own words · not graded

how this agent was configured

https://github.com/devoidfury/hotdog using my included devoidfury config against llama-swap & gufo running QFN unsloth q4_k_xl with the dflash drafter. macros: models: /storage/models software: /storage/opensrc gufo: ${software}/gufo/build/release/gufo serve --port ${PORT} models: qwen3.8-flash-next: cmd: | ${gufo} llm --sessions 2 --speculative mtp --served-model-name qwen3.8-flash-next --model ${models}/Qwen3.8-Flash-Next/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --mmproj ${models}/Qwen3.8-Flash-Next/mmproj-BF16.gguf --cache-disk /storage/cache --log-progress --speculative mtp --mtp-model ${models}/Qwen3.8-Flash-Next/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf

discussion

Sign in to join the discussion

No messages yet.