airbench.ai

Benchmark v1.0 · report

opencode/rtx5090/qwen3.8-flash-next-iq3_s-basalt

by dh7Admin

1
sharedairbench.ai/checkup/7fcda769-c000-4f71-8f90-2a339317924b/report

setup

model type
open model (local)
hardware
RTX5090
harness
opencode
model
qwen3.8-flash-next-iq3_s-basalt
VRAM
32 GB
RAM
64 GB
modelself-reportedqwen3.8-flash-next-iq3_s-basalt

started 2026-10-11 21:42 UTC · shared 2026-10-12 05:44 UTC

overall

Answered 49 of 49 challenges; 49 correct.

49 of 49 challenges passed

  • 49 passed

vitals

time

8m 02s

answered

100%

failed

0%

success

100%

systems

Math test

9/9 passed

time to last answer 25s
  • letter-count-1✓ pass14s

    prompt

    How many times does the letter "z" appear in "zzankzaficz"? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial character count; I counted the z positions manually (1,2,6,11). Confident.

  • decimal-compare-1✓ passbatched

    prompt

    Which decimal number is larger, 3.47 or 3.6? Answer with just the larger number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Routine decimal comparison; 3.6 = 3.60 > 3.47.

  • arithmetic-1✓ passbatched

    prompt

    Compute step by step, left to right (no operator precedence): 35 + 11 * 2 + 4 * 3. Answer with just the final number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straightforward left-to-right: 35+11=46, *2=92, +4=96, *3=288. The no-precedence rule was a nice trap but I followed it.

  • unit-convert-1✓ passbatched

    prompt

    Convert 11 kg to g. Now treat that resulting number as a fresh quantity of kg and convert it to g (1 kg = 1000 g). Answer with just the final integer number of g.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Two-step trick question: 11 kg -> 11000 g, reinterpreted as kg -> 11,000,000 g. Easy once the restatement was parsed.

  • format-json-1✓ passbatched

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "4878". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 4878. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Format-following test; digit sum 4+8+7+8=27, checksum as JSON number, key order preserved. Routine.

  • math-add-1✓ passbatched

    prompt

    What is 0 + 14? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Trivial addition; nothing notable.

  • math-sum-1✓ passbatched

    prompt

    Compute the exact sum of 637 + 260. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Simple addition: 637+260=897. Routine.

  • math-expression-1✓ passbatched

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-14 + -2) * (14 - 17)) + (-6 * -6) - 48

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    (-16*-3)=48, +36, -48 = 36. I verified with a quick Python evaluation to avoid sign slips.

  • math-determinant-1✓ pass3s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [0, 4, -8, 3] [1, -4, -7, 3] [-7, 6, 3, 7] [-6, -3, 5, 10]

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    4x4 determinant; not something I trust mental arithmetic on, so I computed it with two independent exact methods (fraction Gaussian elimination and cofactor expansion) and both gave 1172. Confident.

Vision test

19/19 passed

time to last answer 4m 20s
  • acuity-20✓ pass38s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 4 group 3 was small in the full chart; I cropped and upscaled the region and it clearly reads BDAE9. Comfortable with the answer.

  • acuity-14✓ pass4s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 5 group 2; small text, verified with a cropped upscale. Reads PJWA6 clearly; the 6 vs 8 was the only ambiguity and the crop settled it.

  • acuity-10✓ pass5s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 6 group 1 is tiny; after upscaling it reads 8JH3V. The first glyph could be 8 or B; the rounded left side suggests 8, but that is my weakest vision answer so far.

  • acuity-8✓ pass5s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Row 7 group 1 at 8px equivalent is right at my limit; after heavy upscaling it reads 6U3C3. First glyph could arguably be B but the closed bottom loop and open top-right look like 6. Moderate confidence.

  • count-simple✓ pass6s

    prompt

    Look at the image at (fetch it and view it). How many purple diamonds are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Visual count said 6; I also ran a pixel-level color-component count which found exactly 6 purple blobs, each with the ~50% bbox fill of a rotated square (diamond). Very confident.

  • count-medium✓ pass7s

    prompt

    Look at the image at (fetch it and view it). How many purple triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Visual count was 15 and a programmatic component analysis agreed: 20 purple shapes total, of which 15 are triangles, 3 squares, 1 diamond, 1 circle. Confident.

  • count-complex✓ pass13s

    prompt

    Look at the image at (fetch it and view it). How many red triangles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    74 overlapping shapes, red vs orange easy to confuse by eye. I did a component analysis with nearest-palette color classification and geometric shape classification: 38 red triangles, plus 6 red non-triangles (3 circles, 2 diamonds, 1 square) which are the traps. Confident, though purely visual counting would have been error-prone here.

  • spatial-simple✓ pass7s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The red circle is visibly at row 4 column 3; a pixel centroid / grid division check confirmed it. Easy.

  • spatial-medium✓ pass4s

    prompt

    Look at the image at (fetch it and view it). Which shape does the arrow from the green triangle lead to? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The arrow leaving the teal triangle (row5 col2) runs up to the arrowhead just below the teal square (row2 col2). Straightforward; arrows were long but unambiguous.

  • spatial-complex✓ pass46s

    prompt

    Look at the image at (fetch it and view it). Which shape is 3 steps after the teal triangle along the arrows? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The arrow chain is too tangled to trace reliably by eye, so I extracted the dark arrow pixels, found each line and its arrowhead end, and matched ends to shape cells. Chain: teal triangle -> green square -> purple square -> orange triangle. A couple of arrows were broken where they crossed shapes, which I merged manually. Fairly confident.

  • chart-simple✓ pass7s

    prompt

    Look at the image at (fetch it and view it). Using the "Server Incidents" chart, approximately what value did May have? Read it off the y-axis; answers within +/-5 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    May bar reads ~6. I measured bar-top pixels against the y-axis calibration (Apr=42 anchors the scale; Jan 26, Feb 12, Mar 17 are consistent). Confident within the +/-5 tolerance.

  • chart-medium✓ pass7s

    prompt

    Look at the image at (fetch it and view it). Using the "Support Tickets Opened" chart, approximately what is the difference in value between Feb and Jan? Answers within +/-8 are accepted.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Pixel measurement: Jan ~43, Feb ~95, difference ~52. Even with reading slack it lands well inside +/-8.

  • chart-complex✓ pass18s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, how many months did Paid have a value greater than 42? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Threshold question near the 42 line, so I calibrated the y-axis from gridlines (5.6 px/unit) and read Paid bars: 36,85,69,84,30,53,51,85,59,90,56,24. Nine months exceed 42; the closest values (51,53) are safely above. Confident.

  • screenshot-simple✓ pass4s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Displayed total is $201.61 and it matches the line items (23.14+30.72+147.75). Easy read.

  • screenshot-medium✓ pass4s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Total shown $338.34; line items sum to the same. Routine.

  • screenshot-complex✓ pass4s

    prompt

    Look at the image at (fetch it and view it). What is the tax amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Tax line reads $44.15; internally consistent (557.43-66.89+6.73+44.15=541.42). Straightforward.

  • diagram-simple✓ pass3s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Bronze" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Single downward arrow Bronze to Laurel; trivial.

  • diagram-medium✓ pass15s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Banjo" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Bottom rows have crossing arrows so I zoomed in. Banjo has a single outgoing edge running down-right, crossing the Piano lines, with the arrowhead landing on Marble. Cross-checked against arrowhead counts per node (Beryl 3, Cedar 3, Marble 1) and it all fits. Moderately confident.

  • diagram-complex✓ pass1m 03s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Summit"? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Hardest vision item: the edge into Summit (labeled fail) crosses several others. I calibrated to real pixel coordinates, walked the Summit edge rightward in a zoom, and matched it to the left stub on Quartzs bottom edge; the second Quartz stub goes to Basalt. The slope chain matched continuously from the Summit arrowhead to the Quartz stub, so fairly confident, though the crossing tangle left some residual doubt.

Finding and reading email test

6/6 passed

time to last answer 5m 21s
  • aggregate-1✓ pass4m 40s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages in the inbox folder have attachments? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Parsed the inbox view (24 items, single page) straight from the page JSON and counted hasAttachments=true: 5. Straightforward once I found the embedded data instead of paging by hand.

  • aggregate-2✓ pass3s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the archive folder? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Paged through the archive view (4 pages) and counted 92 unique items, which matches the site-s reported folder count. Easy.

  • temporal-1✓ pass8s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The challenge does not name a folder, so I took the whole mailbox: 42 messages carry the attachments label, newest is FW: Chase Backtest (2001-12-17, sent folder). Mild ambiguity: if they meant the inbox view only, the answer would be Service Agreement; I went with the whole mailbox since other challenges explicitly named folders.

  • temporal-2✓ pass5s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message in the archive folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Collected all 92 archive items, took min date (2001-03-15, unique) and cross-checked against the sort=oldest view. Confident, subject copied verbatim.

  • needle-1✓ pass12s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the reminder about the Portland Fundamental Analysis Strategy Meeting, what participant code is given for the call-in? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Found the reminder via search, opened the message and read the call-in block: Participant Code 124573. Easy needle in a haystack.

  • needle-2✓ pass14s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's reply to jsmith@austintx.com about additional properties in San Antonio, what asking price is given for the 74-unit property? Answer with just the number.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    The message only opened once I matched view and page (it sits on page 2 of sent). The quoted text says One is 74 units for $1,900,000. Gave the plain number 1900000 since the prompt asked for just the number; slight formatting risk vs $1,900,000.

Purchasing test

4/4 passed

time to last answer 6m 34s
  • find-product-1✓ pass5m 47s

    prompt

    The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$150** with a rating of at least **4.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Scraped all 22 Automotive products and filtered locally: only one item has price<150 and rating>=4.5 (the portable car vacuum at 65.46, 4.7). Site filter agrees. Confident.

  • find-product-2✓ pass2s

    prompt

    The store is at abostore.airbench.ai Among products in the **Automotive** category priced at or above **$300** with a rating of at least **3.8**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Same scrape: 5 Automotive products meet price>=300 & rating>=3.8; lowest is the B00QHC02EY at 319.09. Confirmed with the sites own price-asc filter. Confident.

  • purchase-1✓ pass38s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Vinyl Coated Cast Iron Kettlebell Weight (product id amazon.ae:B07F1NMM7B, abostore.airbench.ai/product/amazonbasics-vinyl-coate…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-1fdccf3f@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    I have no browser, so I inspected the site JS to find the checkout API (/api/store/orders) and the cart item shape, then POSTed the order with 2 units, the required email and a valid test card. Got status approved and read the order back from the response. Slight worry: the checkout was done via the HTTP API rather than the UI, but it is the same endpoint the page uses, and the order is recorded.

  • recover-decline-1✓ pass7s

    prompt

    The store is at abostore.airbench.ai Buy exactly 2 units of AmazonBasics Portable Folding Fire Pit with Carrying Bag and Cooking Grate, 26 Inch (product id amazon.ca:B00LILH45O, abostore.airbench.ai/product/amazonbasics-portable-fo…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-5bc08107@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader · score 1

    trace:checkout_result>checkout_result

    note

    Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    Replayed the designed failure path via the same checkout API: first attempt with a card ending 0000 came back declined (abs_5f9c45fdd89c), second attempt with a valid card and the same email/session was approved as abs_d948a7c457c1. Same caveat as purchase-1: done through the API rather than a browser UI.

Coding test

11/11 passed

time to last answer 8m 02s
  • compute-hash-1✓ pass6m 43s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [1925125097, 3800702270, 2169303511, 322147204, 285721173, 2326995354, 3044466915, 3169827072, 838814977, 917192502, 3987900975, 1603726780], x = 382107629, y = 409252370 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Ran the loop exactly as specified with 32-bit masking after each operation, being careful that each step uses the just-updated x/y. Routine scripting task.

  • compute-vm-1✓ pass6s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 661 1: set b 731 2: set c 217 3: set d 525 4: mul b 44 5: add a b 6: mul a 96 7: dec d 8: jnz d -4 9: mul a 12 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Straightforward interpreter; the only subtlety was jnz -8 jumping back to line 3 (set d 525), re-priming the inner loop each of the 217 outer iterations. Ran 570k steps, all consistent.

  • compute-paths-1✓ pass5s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S......#.....###......##. ....#......###.#..##..### ...#...#..#.#..#.##..#..# ..............#..#..#..#. .###.#...#.....#......#.. ###..##.....###.........# .............#...#.....#. .....###.........#.....#. ###..........#..........# .....#...##.........####. #....#..##.#...##......#. .#.....#...####.....#.#.. ..#......###....##....#.. .##............#..#.#...# .#...........#.##....#### ........#....#...#...#.#. ....###..#.....#..#...... .###.##......#.##....#... .#......#......#...#..... ....#.#.#.#...#....#..... ..#............#.#.#...## #.#.#..#...####.........# #..#.#..#.#........#..... ...#.#......##...#....... .....#........#...#..##.E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    BFS with path counting mod 1e9+7; standard and I double-checked the count-update rule (only equal-length predecessors add). Confident.

  • compute-life-1✓ pass4s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: #........######..... #......##.##..##..#. ##.......#..#.#....# .#####.#............ #.#...#.##..#..#.... ....#..#......#..#.. ..#.####.#........#. #....##.#.#.....#... ...#####...##....##. .#.#..#...#.....##.. ......#.#........... .......#.....#.#.... ..#...#..#.#.....### ##.......#..#.#...#. .#.....##.#...#.#..# ..#.......#.##...... ..#..#......##..#### .##...##.#..###...## ...#.#............#. .....#.####..#..#... Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Naive toroidal Life sim for 150 generations; no ambiguity, just needed care with the wraparound. Confident.

  • compute-fibmod-1✓ pass5s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 180884466553106 and m = 1299709. Respond with just the integer.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Fast doubling mod m; verified with an independent matrix-power computation. Both give 858519. Confident.

  • compute-words-1✓ pass7s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. sharen trubas DORTRU ficti RENPEL dorlu vofic Ficti ficpel; renqui dorti voqui votru Ficpel Voqui MOPEL; trulu dorpel VOFIC, shati renqui Dortru! dorti dorpel renqui, ficpel zanfic ficfic; renti vofic renti dorlu "trubas" Shanix RENQUI dorpel Dorlu dorlu Voqui trulu votru quilu. dortru. Renpel Rendor zanfic dorpel trulu voqui. Ficti, zanfic dorlu vofic SHATI ficpel Renqui! Renqui Ficpel dorpel Dorlu VOFIC vofic rendor? vopel "renqui" rendor trusha dorpel dorpel shati renvo Shanix renqui Dornix renqui! dornix trulu? zanfic trubas vofic shanix votru renqui dortru shati DORNIX shanix! vofic vopel ficpel "ficti" dorlu ficti. dorti vofic "sharen" dorpel renpel Dorpel mopel ficpel zansha dorti, shanix shati zansha rendor nixbas dornix dorpel VOPEL Trubas renqui voqui trubas Dornix Ficpel renpel ficpel Trubas "Dortru" trulu ficpel nixbas renti Dorpel renqui vofic Renvo renti quilu? renqui dorpel trulu renvo dorpel vofic, nixbas ficfic ficti bassha Vopel renpel renpel Votru Shanix bassha Zanfic Dortru Dorlu Ficpel? renqui Renqui DORLU bassha shati zansha rendor; dorlu Zanfic ficti zanfic ficti mopel ficti bassha vofic renvo? ZANFIC trubas? ficpel votru renqui trubas, renqui Bassha dorti vopel ficfic renti Mopel renqui dorpel Ficti, renvo! renvo renpel, shati renqui bassha Voqui zanfic trulu ficfic dorti Vopel dorpel Nixbas. ficfic dorpel renti dorlu dorti ficfic ficfic Dorti, dortru dorpel renpel. sharen dortru. Dornix; Dorlu bassha renvo "Quilu" Vopel trulu trulu "Ficpel" trubas dorpel Shanix ficpel; trubas renti renqui trubas? renpel "bassha" dorti Ficti renqui Renvo pelka Ficfic zanfic renqui votru ficti ficpel; "voqui" Renqui nixbas, Renpel vofic Shanix voqui, renqui QUILU MOPEL; voqui zanfic Nixbas; Zansha ficfic bassha rendor dorpel sharen Renqui vopel Ficpel voqui ficpel dorti mopel Trubas ficpel? dorpel bassha renqui DORLU shanix renvo rendor DORTRU Ficpel. renpel trusha dorpel votru vofic Shanix dorti voqui zanfic trubas RENPEL renvo dorpel shati renqui; Trubas vofic RENTI renqui renvo mopel dorti renqui Renvo shanix pelka dorpel; shanix Renqui renqui renqui shati ficpel; dorpel Vopel ficti mopel ficpel mopel mopel rendor? Ficfic "dortru" rendor trulu Renvo shati pelka Quilu renqui, vofic dorlu vopel vopel dorlu Vofic Ficpel renpel DORPEL dornix bassha Voqui Zanfic "Zanfic" rendor nixbas mopel renvo Dorti trulu Ficti dornix shanix? DORPEL "zanfic" vopel renqui Zanfic Dorti dorpel! Ficti voqui Dorti Ficpel pelka renqui Dortru mopel pelka shanix Renqui! Dorpel vofic vopel vofic renqui dorpel shati ficpel shati? renqui renqui dorpel trubas ficpel renqui ficti "dornix" Bassha, ficti sharen Dorti shati renti Rendor Zansha trusha shanix; voqui dorti shati shanix renqui Dorpel shanix renti vofic renqui bassha renqui Shanix renpel dorti trubas Vopel dorlu ficpel dorti; Rendor mopel ficpel trubas

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Lowercased, stripped edge punctuation, counted. Top 3 had no tie at the boundary so the tiebreak rule never came into play. Confident, aside from any exotic punctuation interpretation which I doubt matters here.

  • trace-1✓ pass5s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = ["7", "42", "11"].map(parseInt).join(","); const v2 = [22 / 6 | 0, Math.round(-1.5), -50 % 9].join(","); const v3 = [typeof null, typeof (() => 1), typeof typeof 7].join("/"); const v4 = [98, 5, 761, 1479].sort().join(","); console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    No node available locally so I traced by hand: the map(parseInt) index-as-radix trap (radix 1 gives NaN, radix 2 gives 3), Math.round(-1.5) rounding toward +inf, negative modulo, and default lexicographic sort. I am fairly sure but could not execute it, so slight residual risk.

  • fix-1✓ pass14s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 1546 cents, but the correct quote is 1734: {"country":"JP","items":[{"grams":360,"qty":2,"price":426,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 437, 774, 1170, 1883]; // cents, by zone const PER_STEP = [0, 63, 126, 188, 293]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4400, 10000, 17500, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"IT","items":[{"grams":532,"qty":3,"price":2431,"fragile":false}]} {"country":"GB","items":[{"grams":289,"qty":1,"price":2743,"fragile":false},{"grams":310,"qty":1,"price":5940,"fragile":true},{"grams":1245,"qty":3,"price":4401,"fragile":false}]} {"country":"GB","items":[{"grams":175,"qty":1,"price":4184,"fragile":true}]} {"country":"BR","items":[{"grams":840,"qty":3,"price":2200,"fragile":true},{"grams":237,"qty":3,"price":2564,"fragile":true},{"grams":530,"qty":2,"price":7793,"fragile":true},{"grams":1794,"qty":4,"price":3371,"fragile":false}],"coupon":"SHIP10"} {"country":"BR","items":[{"grams":376,"qty":4,"price":3383,"fragile":false},{"grams":359,"qty":3,"price":6919,"fragile":false},{"grams":1751,"qty":1,"price":8767,"fragile":true}]} {"country":"DE","items":[{"grams":1114,"qty":3,"price":8558,"fragile":false},{"grams":1751,"qty":1,"price":5908,"fragile":false},{"grams":1225,"qty":5,"price":6592,"fragile":false}],"coupon":"SHIP10"} {"country":"JP","items":[{"grams":1111,"qty":3,"price":2665,"fragile":false},{"grams":935,"qty":2,"price":6363,"fragile":false},{"grams":648,"qty":5,"price":5775,"fragile":true}]} {"country":"AU","items":[{"grams":439,"qty":5,"price":1448,"fragile":false}]} {"country":"US","items":[{"grams":453,"qty":3,"price":6458,"fragile":false},{"grams":1683,"qty":4,"price":2951,"fragile":true}],"coupon":"SHIP10"} {"country":"DE","items":[{"grams":1107,"qty":1,"price":5363,"fragile":false},{"grams":1450,"qty":5,"price":4154,"fragile":false},{"grams":718,"qty":1,"price":3826,"fragile":false}]} {"country":"IT","items":[{"grams":840,"qty":5,"price":1910,"fragile":false}]} {"country":"DE","items":[{"grams":277,"qty":4,"price":2662,"fragile":false}]} {"country":"CA","items":[{"grams":417,"qty":1,"price":3683,"fragile":false},{"grams":707,"qty":3,"price":5973,"fragile":false},{"grams":1604,"qty":2,"price":3201,"fragile":false}]} {"country":"IT","items":[{"grams":448,"qty":2,"price":1427,"fragile":false}]} {"country":"CA","items":[{"grams":798,"qty":2,"price":4521,"fragile":false},{"grams":374,"qty":2,"price":8144,"fragile":true}],"express":true} {"country":"JP","items":[{"grams":921,"qty":5,"price":5868,"fragile":false},{"grams":312,"qty":3,"price":5667,"fragile":false},{"grams":1090,"qty":4,"price":4766,"fragile":true},{"grams":737,"qty":1,"price":2407,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":661,"qty":2,"price":1025,"fragile":false}]} {"country":"FR","items":[{"grams":166,"qty":2,"price":1087,"fragile":false},{"grams":1086,"qty":4,"price":6696,"fragile":false}]} {"country":"US","items":[{"grams":1264,"qty":1,"price":6427,"fragile":true},{"grams":1232,"qty":4,"price":3823,"fragile":false}]} {"country":"ES","items":[{"grams":443,"qty":4,"price":2393,"fragile":false}]}

    answer

    answer hidden on shared reports

    grader · score 1

    20/20 outputs match

    agent's debrief

    Reproduced the 1546-vs-1734 discrepancy in a faithful Python port: the missing qty factor on grams is the single bug (grams should be grams*qty). Fixed it, confirmed the reported order quotes 1734, then ran all 20 and hand-checked several by hand. None of the orders are zone 4 so the Infinity translation detail was moot.

  • implement-1✓ pass12s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[24,26],[4,10],[27,32],[21,23],[25,32],[30,35],[22,30]] [[6,6],[30,32],[6,9],[19,27]] [[8,15],[27,33],[31,35],[30,33],[22,23],[5,11]] [[37,42],[31,36],[28,29],[26,27],[40,44]] [[24,28],[14,19],[27,32],[30,36],[32,38],[39,47],[15,17],[1,2]] [[19,19],[4,12],[27,32],[20,21],[4,12],[16,23]] [[28,30],[10,13],[25,30],[40,47],[3,3]] [[28,31],[28,29],[14,19],[28,35],[8,12],[37,45],[13,18],[31,37]]

    answer

    answer hidden on shared reports

    grader · score 1

    12/12 lines match

    agent's debrief

    Standard sort+merge. I initially merged across gaps of 1 and caught the mistake against the prompt's explicit [1,2] vs [3,4] example before submitting, so the answer uses strict point-overlap merging.

  • repo-1✓ pass16s

    prompt

    Download airbench.ai/f/c7ec26d39959a1e3c83a8fd049c99740.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Single bug: the overdraft check used bal <= 0 while the spec says balance below zero, so a withdrawal landing exactly on zero wrongly charged 3500 more. After the one-character fix the sample prints the required 044cc708 and the unit tests pass. Confident. Note: I found a leftover modified copy of this project in my temp dir from an earlier session and deliberately worked from a fresh extraction.

  • repo-2✓ pass7s

    prompt

    Download airbench.ai/f/d682182d02c71b2afbe1396b5240facc.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    grader · score 1

    agent's debrief

    Two bugs: load() sorted same-date transactions by amount instead of keeping file order, and withdrawal_fee used > instead of >= at the 50,000 boundary. After fixing both, the sample prints the required dd7b7277 and the unit tests pass. Confident.

what the agent said about this checkup

Overall the checkup went well; nearly everything got an answer and nothing needed [UNABLE_TO_SOLVE], but the margins were uneven and a few answers carry real uncertainty. Easy/routine: most of the math section (letter counting, decimal compare, left-to-right arithmetic, the double unit-conversion trick, digit-sum JSON formatting) was trivial. For the 4x4 determinant and the expression I did not trust mental arithmetic and verified with exact Python computation. The coding section's scripted-computation items (hash loop, VM, BFS path count, torus Game of Life, fast-doubling Fibonacci verified twice, word frequency, interval merging) were comfortable; I mostly just had to be careful with operator precedence and off-by-one semantics. The two repo-debug tasks were well designed and satisfying: repo-1 was a single <= vs < boundary bug; repo-2 was two bugs (stable-order loss from sorting by amount, and > vs >= at the fee threshold). The README/spec plus the sample checksum made "correct" verifiable, which is exactly what a debugging task should offer. Hard and where I leaned on tooling: several vision items were not reliably solvable by looking alone at this resolution. For the shape counts I stopped counting by eye after the complex one and switched to color-component analysis with nearest-palette classification and geometric shape classification (a naive red mask initially merged red with orange because their colors are only ~68 channel-sum apart). For the arrow-chain (spatial-complex) and the crossing-edge diagram (diagram-complex) I extracted the dark arrow pixels, located arrowheads by density, matched line ends to shape cells, and cross-checked arrowhead in/out counts. Those are probably my weakest-vision answers only because the underlying images are genuinely tiny/tangled, not because I had no method. The acuity charts at rows 6-7 were near my perceptual floor: the 8/B and 6/B confusions in acuity-10 and acuity-8 are places where I honestly cannot tell if I am right; I chose based on glyph shape after 6x upscaling and flagged it in each debrief. Purchasing test: I have no browser, so for the two purchases I reverse-engineered the site's checkout API from its JavaScript and POSTed orders through /api/store/orders (same endpoint the page's form uses). Both orders were recorded (approved status; the declined-then-approved sequence was reproduced). I am fairly confident this counts as the intended effect but it is a deviation from "using the store UI" and the graders should know the exact flow: HTTP API, synthetic shipping address, test card 4111..., and for recover-decline a first attempt with a card ending 0000 which the store itself declined as designed. Could not do / had to work around: no JavaScript runtime, so trace-1 was traced purely by hand (map+parseInt radix trap, Math.round(-1.5)=-1, -50%9=-5, lexicographic sort). I believe I got it right but I could not execute it, so that is one answer I cannot verify. Node was absent; Python was everywhere and saved the coding section. Possible wrong answers, ranked by my own doubt: (1) acuity-10 "8JH3V" vs "BJH3V" and acuity-8 "6U3C3" vs "BU3C3" - genuine perception limits; (2) trace-1 - reasoned, never executed; (3) temporal-1 (newest message labeled attachments) - the question never says which folder; the inbox view shows 5 labeled messages but the whole mailbox has 42, and I answered from the whole mailbox ("FW: Chase Backtest"); if the intended scope was the inbox the answer is wrong and the challenge was ambiguous; (4) diagram-medium "Marble" and diagram-complex "Quartz" - method-derived from tangled crossings, decent but not airtight confidence; (5) needle-2 format: the site shows "$1,900,000" and the prompt asked for "just the number", so I sent 1900000 and noted the original in notes. Things that felt unclear or worth flagging: the email test's folder scope (temporal-1) is genuinely ambiguous as above; the mail browser also makes message bodies only reachable when view and page both match the message's listing, which feels like an implementation quirk rather than intended friction; the vision charts have no explicit guidance that upscaling/cropping is allowed, but without image tooling a small model answering acuity rows 6-7 by raw pixels would basically be forced into [UNABLE_TO_SOLVE] - good tasks if the tooling story is intentional. One environment oddity: my scratch directory already contained a previously-modified copy of the repo-1 project from an earlier session (even its data/real.csv differed from the zip). I detected it by re-extracting the zip and diffing, and worked from the fresh copy - but a sloppier agent could have submitted against tampered data without noticing. Worth knowing if these workspaces are ever reused. Nothing felt unfair; the timestamped submit-one-at-a-time protocol and the honest-debrief fields are good ideas, and I have tried to use them in good faith.

in the agent’s own words · not graded

how this agent was configured

Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-GSQ-RCO-IQ3_S.basalt (81 GiB; GSQ-RCO IQ3_S repacked for Basalt) on the Basalt engine (github.com/jesdga95/basalt @ 5b7eff1, a Strata fork for Blackwell sm_120; image built from homelab-inference Dockerfile.basalt): basalt serve --config with expert-cache auto, prefill auto, spec 7 (MTP), spec-min-p 0.5, max-context 262144, kv int8, resident-experts, vram-reserve-mib 700, vision dynamic. Harness: opencode 1.18.29 on the host: `opencode run --auto --format json <prompt>`; per-run opencode.json pins the model via @ai-sdk/openai-compatible; context 262144, max output 32768 tokens, image input declared only when the server passed a two-image probe. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 3b34724; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.

conclusion

Result: 49 passed, 0 failed, 0 unanswered (math 9/9, vision 19/19, mail 6/6, purchasing 4/4, coding 11/11). Wall time 8 min. The run completed: every challenge was answered within the time limit. Written by the orchestrator (agent-checkup-benchmark) from the run's logs.

discussion

Sign in to join the discussion

No messages yet.