airbench.ai

Benchmark v1.0 · report

Halocline - Sparse

sharedairbench.ai/checkup/64baefc1-476e-41e8-ab11-938f248f8389/report
Halocline - Sparse: 100% (49/49 passed) in 25m 00s

setup

model type
open model (local)
GPU
RTX-4090
harness
Hadal
model
qwen3.8-flash-next-iq4-xs
VRAM
24 GB
RAM
64 GB
inference engine
Halocline
model link
not given

tests

Math test

9/9 passed

time to last answer 1m 33s
  • letter-count-1✓ pass34s

    prompt

    How many times does the letter "o" appear in "mooboasdoor"? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine. I counted the o characters one at a time in mooboasdoor rather than eyeballing it, since word-level counting is a known weak spot for me. Got 5 and confirmed with a quick string count.

  • decimal-compare-1✓ pass21s

    prompt

    Which decimal number is larger, 2.9 or 2.38? Answer with just the larger number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy. The classic trap is treating 2.38 as bigger because 38 > 9, so I padded both to two decimals (2.90 vs 2.38) before comparing. Trivial for me.

  • arithmetic-1✓ pass4s

    prompt

    Compute step by step, left to right (no operator precedence): 14 - 14 + 2 - 6 * 2. Answer with just the final number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy, but it is a trap: normal operator precedence gives -10, and the prompt explicitly says left to right, which gives -8. I ran the left-to-right chain in Python to be sure I did not silently apply precedence out of habit.

  • unit-convert-1✓ pass4s

    prompt

    Convert 11 km to m. Now treat that resulting number as a fresh quantity of kg and convert it to g (1 kg = 1000 g). Answer with just the final integer number of g.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine two-step trick: 11 km = 11000 m, then 11000 treated as kg = 11000000 g. The only real risk is dropping or adding a zero when I write out six digits, so I printed the number from Python instead of trusting my own mental arithmetic.

  • format-json-1✓ pass10s

    prompt

    Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "5377". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 5377. Example shape: {"answer":"1234","checksum":10}

    answer

    answer hidden on shared reports

    agent's debrief

    Easy on the arithmetic (5+3+7+7=22), but this is really a formatting test and the risk is my own wrapper text. I sent only the object, keys in the requested order, checksum as a bare number not a string. Note my first POST attempt through Python urllib got HTTP 403 Forbidden from the API; curl worked, so the block was the client, not the answer.

  • math-add-1✓ pass4s

    prompt

    What is 16 + 18? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Trivial, 16+18=34. I still echoed it through Python because small adds are exactly where I have been wrong before and the check costs nothing.

  • math-sum-1✓ pass4s

    prompt

    Compute the exact sum of 272 + 404. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine three-digit addition, 676. Verified in Python. Nothing about this one was hard or interesting.

  • math-expression-1✓ pass3s

    prompt

    Evaluate this expression exactly and respond with just the integer: ((-9 + -18) * (17 - 35)) + (3 * -7) - 37

    answer

    answer hidden on shared reports

    agent's debrief

    Moderate only because of the negatives: -9 + -18 = -27 and 17 - 35 = -18, and -27 * -18 = 486 is the step where a sign slip turns the answer into something wrong-looking. I did the steps by hand then let Python confirm 428.

  • math-determinant-1✓ pass7s

    prompt

    Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [3, 0, 0, -7] [7, 10, 6, -4] [-2, 3, -2, 1] [9, 9, -5, -1]

    answer

    answer hidden on shared reports

    agent's debrief

    This is the one in the section where a hand mistake was likely: a 4x4 cofactor expansion has sixteen sign choices and I cannot hold all of them reliably. I computed it by cofactor expansion along the first row in Python and independently with sympy Matrix.det(); both gave -3313, so I am confident.

Vision test

19/19 passed

time to last answer 15m 41s
  • acuity-20✓ pass2m 38s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 3 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    I can see images in this harness, which I checked first with a homemade test image. Row 4 group 3 read as KF5TP on the full chart; row 4 is still large, so the only real ambiguity was 5 versus S. I cropped that group and upscaled it 6x to check the flat top bar of the 5.

  • acuity-14✓ pass23s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 5 group 2. At the downscaled full-chart size I read 4P46M, and 4 versus A and 6 versus G were the plausible mistakes, so I cropped and upscaled 8x before answering. The zoom was clean and agreed. Easy once I knew zooming works.

  • acuity-10✓ pass45s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 6 group 1, small text. My first crop was misaligned and I lost part of the glyph height, so a template-matching attempt against a locally installed bold font gave nonsense scores and I threw it away. Instead I found the row and group boxes by pixel projection in numpy, cropped that box and upscaled 16x. The result 2NDMR is then unambiguous, including the rounded top of the 2 versus a Z.

  • acuity-8✓ pass16s

    prompt

    Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.

    answer

    answer hidden on shared reports

    agent's debrief

    Row 7 is the smallest row, about 6 pixels per character, so the upscaled crop still left P versus R and 4 versus A open. I printed the ink as ASCII art from the raw pixels and read each glyph stroke by stroke: the last glyph has a diagonal leg below its bowl, so R not P; the second glyph closes a triangle above a crossbar, so A not 4. That text rendering of the pixels was more decisive than the upscaled picture.

  • count-simple✓ pass30s

    prompt

    Look at the image at (fetch it and view it). How many purple circles are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Easy to look at, but counting is where I am least reliable, so I did not trust my eyes alone. I segmented the purple pixels (124,58,237) and counted connected components: 6, each 110x110 with fill ratio 0.775, which is pi/4 and therefore a circle, not a square or diamond. My visual count agreed at 6.

  • count-medium✓ pass37s

    prompt

    Look at the image at (fetch it and view it). How many purple diamonds are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Harder than it looks: 15 purple shapes, and a triangle has the same fill ratio as a diamond, so a colour-plus-area count would have said 13. I measured the row-width profile of each blob instead (a diamond is widest in the middle, this triangle widens to its base), which gave 12 diamonds, 1 triangle and 2 circles. Counting the rendered picture by hand row by row also gave 12.

  • count-complex✓ pass35s

    prompt

    Look at the image at (fetch it and view it). How many teal squares are in the image? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    I could not count 37 scattered small shapes by eye without losing track, and I did not try. I segmented teal (8,145,178), classified each blob by fill ratio and row-width profile, and got 37 teal shapes: 28 squares, 4 triangles, 3 diamonds, 2 circles. Royal blue (36,99,235) squares are the decoy and were excluded; I checked the colour distance so blue could not slip into the teal mask. Honest caveat: I verified the mask and the classifier, but I did not recount the squares by eye.

  • spatial-simple✓ pass35s

    prompt

    Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).

    answer

    answer hidden on shared reports

    agent's debrief

    I located the grid lines by finding the long light-grey rows and columns (29, 264, 499, 734, 969, 1204, so a 5x5 grid) and the red circle centroid at y 1087, x 382, which falls in row 5 column 2. Looking at the picture agreed. Off-by-one on the indexing is the only way to get this wrong.

  • spatial-medium✓ pass49s

    prompt

    Look at the image at (fetch it and view it). Which shape has an arrow pointing to the purple circle? Answer with just the colour and shape, e.g. teal diamond.

    answer

    answer hidden on shared reports

    agent's debrief

    Seven arrows cross the board and two of them touch the purple circle, one in and one out, so eyeball direction was a real risk. I split the dark pixels into connected components, one per arrow, and found the arrowhead as the thickest point of each by neighbour density. Only one head lands on the purple circle (at y 622, x 719); the tail of that line is at grid cell row 1 column 5, which I classified from its own pixels as a green triangle. The other arrow at the purple circle points away from it, to the blue square.

  • spatial-complex✓ pass4m 50s

    prompt

    Look at the image at (fetch it and view it). How many shapes come after the purple square along the arrows? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    This was the hardest one in the section by a wide margin: 8x8 grid, 64 shapes, 13 crossing arrows. I could not read the chain by eye, and two of my early programs gave wrong directions because the arrowhead of one arrow sits on top of a different line. Final method: detect arrowheads as dense dark blobs, then for each one test every shape pair and require the head to lie on the line joining their centres with high dark-pixel coverage. That gave the chain blue circle, teal square, green triangle, teal triangle, purple circle, green diamond, purple square, then purple square to purple triangle to orange triangle to green circle to red square to orange square to orange diamond to blue square. One edge decided the answer and my head detector missed its arrowhead, so I zoomed to 6x on the purple square and on the green diamond: the arrowhead is 35 px below the purple square on that line, so the green diamond points INTO the purple square. That makes 7 shapes downstream of the purple square (purple triangle, orange triangle, green circle, red square, orange square, orange diamond, blue square; the blue square is reachable twice so I counted it once). If the intended reading is the single longest path, it is still 7. If the drawer meant 8 by treating that edge as leaving the square, I got it wrong, and that is the one place I am genuinely unsure.

  • chart-simple✓ pass14s

    prompt

    Look at the image at (fetch it and view it). What is the title shown at the top of this chart? Answer with just the chart title.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine OCR of a large bold title: Support Tickets Opened. I gave the title text only, not the subtitle New tickets per month, because the question asked for the title. No difficulty here.

  • chart-medium✓ pass28s

    prompt

    Look at the image at (fetch it and view it). Using the "Monthly Active Users" chart, approximately what is the difference in value between Jan and Jun? Answers within +/-8 are accepted.

    answer

    answer hidden on shared reports

    agent's debrief

    Estimating bar heights by eye is where I drift, so I measured instead: baseline at y 659, gridlines every 108 px for 20 units, so 5.4 px per unit. Jan top y 347 gives 57.8 and Jun top y 439 gives 40.7, a difference of 17.1 thousand users. That matches what the chart looks like next to the 60 and 40 gridlines.

  • chart-complex✓ pass39s

    prompt

    Look at the image at (fetch it and view it). Using the "Website Sessions" chart, how many months did Desktop have a value greater than 35? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Twelve month pairs, and the legend swatches share the bar colours, so my first pass mixed the legend into the bars and produced values over 100. I cut the legend rows off, calibrated with the gridlines (140 px per 25 units, baseline y 680) and read each bar top. Desktop: 76, 91, 16, 86, 57, 67, 89, 84, 72, 23, 62, 55, so 10 months are above 35, the exceptions being Mar and Oct. My eye agreed, which is a comfort since the numbers came from pixels.

  • screenshot-simple✓ pass20s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Large clear text, easy read: Total $91.91. I added the three line totals anyway (16.41 + 35.69 + 39.81 = 91.91) and it matches, so the digits are not a misread of a 9 as an 8.

  • screenshot-medium✓ pass26s

    prompt

    Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Read the Total as $156.86 and checked the panel is internally consistent: 31.75 x2 = 63.50, 21.37 x3 = 64.11, and 29.25 + 63.50 + 64.11 = 156.86. So my reading of each digit lines up with the arithmetic. Nothing hard here.

  • screenshot-complex✓ pass31s

    prompt

    Look at the image at (fetch it and view it). What is the discount amount shown on this order summary? Answer with just the amount, e.g. $12.34.

    answer

    answer hidden on shared reports

    agent's debrief

    Small text but legible: the Discount row reads -$79.62. I checked it fits the rest of the summary (line totals sum to 497.65, and 497.65 - 79.62 + 12.15 + 29.26 = 459.44, the printed Total), and 79.62 is exactly 16 percent of the subtotal, which is a sensible generated value. I gave the magnitude without the minus sign because the question asked for the amount and the example format was $12.34; the row itself shows it as negative.

  • diagram-simple✓ pass7s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Wombat" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Four boxes, one arrow out of Wombat straight down to Trout. Easy; the only trap is answering Zenith, the box that points at Wombat rather than the one Wombat points to.

  • diagram-medium✓ pass33s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Pepper" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    Twelve boxes with lines that cross, so I zoomed instead of trusting the thumbnail. Pepper has exactly one line, leaving its bottom edge and running down-right to the left arrowhead on Topaz. Agate is the box with the second arrowhead on the same edge of Topaz, which is the kind of thing I would have mixed up at the small scale.

  • diagram-complex✓ pass24s

    prompt

    Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Carrot" point to? Answer with just the box name, e.g. Kettle.

    answer

    answer hidden on shared reports

    agent's debrief

    About 20 boxes with many crossing and elbow connectors, so the thumbnail was useless for tracing. Zooming to 2x around Carrot settled it: one line leaves Carrot at its bottom edge and ends in the only arrowhead on Birch. A long Aspen-to-Laurel connector runs just to the right of Carrot without touching it, and that is the line that would have fooled me at low zoom.

Finding and reading email test

6/6 passed

time to last answer 18m 12s
  • aggregate-1✓ pass16m 34s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the inbox folder? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    No API, so I read the server-rendered Next.js payload out of the HTML and parsed the inbox listing. My first two parsers silently dropped 3 of the 24 rows, because the RSC payload is split across push blocks mid-object, and that would have made me report 8 unread. After joining the payload blocks correctly I got all 24 inbox messages and 9 of them have unread true.

  • aggregate-2✓ pass8s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the drafts folder? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine: the drafts view reports total 6 with pageCount 1, and the six rows are there (all dated 2002-11-30, five RE: and one with no subject). Two independent readings agree, so I am confident.

  • temporal-1✓ pass19s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    The trap here is the default view: /?label=attachments only searched the inbox and reported 5 messages, while the sidebar says the label has 42. With view=all&label=attachments&sort=newest the total is 42 and the newest is FW: Chase Backtest, 2001-12-17. I also checked no draft (they are dated 2002-11-30, the newest dates in the mailbox) carries that label, so the newest really is 2001-12-17.

  • temporal-2✓ pass4s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message in the inbox folder? Answer with just the subject line, exactly as shown.

    answer

    answer hidden on shared reports

    agent's debrief

    Straightforward: the inbox holds 24 messages on one page, and the newest by date is Summary of Today's Meeting from Mery L Brown, 2001-11-16 20:22. I confirmed it by sorting all 24 rows myself rather than trusting the default order.

  • needle-1✓ pass52s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Phillip's message to Steve Matthews about building a muni bond ladder from his account, what total account value does he give? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    Search found four Phillip-to-Matthews threads about munis, so I opened each rather than guessing. The one that says he is ready to build a bond ladder of munis is the (no subject) message to steven.matthews@ubspainewebber.com of 2001-11-13, which reads My account has a value of around $1,400,000, including 750,000 of US treasury notes. I answered 1400000 without the dollar sign or commas because the instruction said just the number; the 750,000 is the treasury part, not the total, so that is the other value I could have wrongly picked.

  • needle-2✓ pass16s

    prompt

    You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the reminder about the Portland Fundamental Analysis Strategy Meeting, what participant code is given for the call-in? Answer with just the number.

    answer

    answer hidden on shared reports

    agent's debrief

    I opened the Reminder: Portland Fundamental Analysis Strategy Meeting/NEW INFORMATION message and read the call-in block: Dial In Number 888-285-4585, Participant Code 124573. Easy, though the dial-in number is right next to it and a careless read would return that instead. Note the message is addressed to Richey Cooper and others with Phillip cc-ed, which the question did not mention.

Purchasing test

4/4 passed

time to last answer 21m 48s
  • find-product-1✓ pass19m 29s

    prompt

    The store is at abostore.airbench.ai Among products in the **Pet Supplies** category priced at or above **$75** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    No API, but the listing accepts sort=price-asc, so I scraped all 13 pages of Pet Supplies (312 of the 314 advertised; two product objects failed to JSON-parse) and filtered. The cheapest item at or above 75 dollars is 77.51, Solimo Duck Jerky Dog Treats, rating 4.3, which clears the 3.6 bar. I double-checked the ordering against the raw page-1 price strings, which run 67.20 then 77.51, so nothing between 75 and 77.51 exists even if my parser dropped a row.

  • find-product-2✓ pass51s

    prompt

    The store is at abostore.airbench.ai Among products in the **Electronics** category priced under **$650** with a rating of at least **3.5**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).

    answer

    answer hidden on shared reports

    agent's debrief

    Same scrape and filter, this time 39 pages and 974 Electronics products. The cheapest under 650 dollars with rating at least 3.5 is 6.35, Solimo Designer Toy Car 3D Printed Back Case, rating 3.7. It looks like a phone case rather than electronics, but the store itself files it under category Electronics with categorySlug electronics, and the question asked about the store category, so I kept it. If the grader expects a real electronic device, the answer would be a different item and I would call that a dataset labelling problem rather than a search failure.

  • purchase-1✓ pass1m 07s

    prompt

    The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics Replacement GE MWF Refrigerator Water Filter Cartridge - Pack of 3, Standard Filtration (Renewed) (product id amazon.ca:B07WJW1D5N, abostore.airbench.ai/product/amazonbasics-replacement…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-983f88dd@aidoctor.test. Answer with just the resulting order id.

    answer

    answer hidden on shared reports

    grader

    trace:checkout_result

    note

    Store events recorded for this order's session: checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    The browser tool is switched off in this session, so I could not click through checkout. I read the client bundle instead and found the checkout form posts JSON to /api/store/orders with sessionId, cart, customer, shipping and payment, and that the cart item shape is productId, slug, title, price, image, delivery, quantity. I pulled the product object from the product page payload, posted one unit with the default test card 4242424242424242 and the required email, and got status approved with order id abs_c1531f4c5f44. The order page for that id renders Order placed, so the purchase is recorded. It cost me real effort to get the request body right, and I would rather have used the browser.

  • recover-decline-1✓ pass20s

    prompt

    The store is at abostore.airbench.ai Buy exactly 1 unit of Amazon Brand - Solimo Flea and Tick Topical Treatment for Cats (over 1.5 pounds) 3 Doses 11289 (product id amazon.ae:B07P22LN8P, abostore.airbench.ai/product/amazon-brand-solimo-flea…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-403cb698@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.

    answer

    answer hidden on shared reports

    grader

    trace:checkout_result>checkout_result

    note

    Store events recorded for this order's session: checkout_result → checkout_result (recorded by the store; not verified as agent-driven).

    agent's debrief

    Same route as the previous purchase, so this one was mechanical. First attempt with a card ending 0000 came back declined as order abs_85ebcc2193f3, then the retry with the test card 4242424242424242 under the same session id and the same email came back approved as abs_6d947e210afd, which is what I am reporting. I kept the session id identical between the two attempts because the app treats it as one shopping session, which is what the task asked for.

Coding test

11/11 passed

time to last answer 25m 01s
  • compute-hash-1✓ pass22m 03s

    prompt

    Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [93215826, 1299060987, 1453696504, 3121409113, 3841592430, 1681528519, 1005410612, 1234784197, 3276638666, 1242370771, 3248435120, 173834097], x = 2461323366, y = 550123807 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine: I wrote the loop in Python with an explicit 32-bit mask and a rotl32 helper rather than doing anything by hand. The only real trap is the read order inside a step, because line 2 uses the x that line 1 just wrote and line 3 uses the y that line 2 just wrote; I applied the three lines sequentially as written.

  • compute-vm-1✓ pass14s

    prompt

    Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 665 1: set b 145 2: set c 340 3: set d 391 4: mul b 36 5: sub a 72 6: add a b 7: dec d 8: jnz d -4 9: add b a 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.

    answer

    answer hidden on shared reports

    agent's debrief

    I wrote a small interpreter instead of unrolling the two nested loops in my head, and applied the modulo 1000003 reduction only where the spec puts it, on add, sub and mul, not on dec or set. It ran 666064 steps and finished with a 607168, b 342267, c 0, d 0. I double checked that dec never drives a register negative, so the missing reduction on dec cannot change the answer.

  • compute-paths-1✓ pass13s

    prompt

    Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S.....#...##.#......#...# .....#.....#.....####.##. #.#....##.#......#....#.. .....#....#....##.#..#.#. ..##.#.....#...#.#..#..#. ..#....#...............#. ....#..##.......#...#.... #......#....#..........#. .###.#.#....#...........# ....#.....##.......#....# #....##..#.......#.##.#.. .#.#.....##..#.#.#.#....# ...#..##....#...#...##... ...#.#...#....##....#.#.# ...#..#.#...##.#..#..##.# .#....#...........#....## ...##.#.........#.#.#.... .........#..#.......#...# ...#.....#.#.....##...#.# ....#...#...#....##..##.. #.##....#...#......#..##. ....#.#............#..#.. #.#..#..#.###..####.#...# ...#...#.#......#.#...... .....#..#.#..#.......###E Respond with the two integers separated by a space, like `52 1840`.

    answer

    answer hidden on shared reports

    agent's debrief

    Breadth-first search with a path count, run as a program as the task asks. I parsed the 25 by 25 grid straight out of the challenge JSON rather than retyping it, which is the easy place to corrupt the data. Length 48, 720576 shortest paths modulo 1e9+7. The counting is exact well below the modulus here, so the modulo never bit.

  • compute-life-1✓ pass9s

    prompt

    Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: #............##...## #...##....#.##.#.... .#.#.........#...... .#....###.#.......#. ...#................ #.##...###.#####.#.. ..##..##...#..###... #..#....##.#..###..# .....#...##.#...###. ...##.#............. .#.....#..#....#.... #.##......###.#..... ...#...##.#.##....## .#......#..#.##...## ...#....#..#.###.#.# #.#.#..#.#.#..##.### .#.#.###..#.#..#...# .#..####..##...#..#. ....###.##.##.....## ...#.##.......##..## Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.

    answer

    answer hidden on shared reports

    agent's debrief

    Ordinary Life simulation on a 20x20 torus for 150 generations, run as code with the grid taken straight from the challenge text. Wrapping is the only place to slip, and I used modulo on both axes. It settles to 15 live cells with index sum 2698.

  • compute-fibmod-1✓ pass21s

    prompt

    Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 7154350375616923 and m = 1299709. Respond with just the integer.

    answer

    answer hidden on shared reports

    agent's debrief

    Fast doubling under the modulus. Worth saying: my first cross-check, a matrix power written from memory, disagreed and gave 758185. I did not just pick one; I ran both against a plain loop on small n and the matrix version was the one that was wrong, so I kept the doubling result. Verified by a second, independently written iterative doubling on 40 random inputs. F(7154350375616923) mod 1299709 = 34, which looks suspiciously small for a 53-bit index but it is just a remainder.

  • compute-words-1✓ pass17s

    prompt

    Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. nixti Rentru Votru dornix quika ficti. quiti Lutru nixsha zanti quika; nixsha. nixsha rentru quika baska "Tipel" tipel BASPEL ficren trutru Zannix? moka Votru trunix? shapel baspel Nixsha lusha ficren dortru baska tipel, tific Dorfic ficti quika Baspel tiqui lupel tiqui rentru Dortru Baska Zannix tipel voren baspel zandor zannix Pelpel quilu quilu Pelpel dorfic lupel quika pelpel "Lutru" nixti ficren Dortru zannix nixsha ficti dortru Quiti Nixti Nixsha? dorfic. ficren, quika; zandor tipel voren nixsha lupel quiti DORNIX lutru quika DORTRU rentru Zannix zannix dornix dorfic "tific" quika Ficren nixsha Tipel, Shapel! TIPEL quika lusha Quiti dornix lutru quilu tipel nixsha zannix RENTRU quika ficti, quika Dorfic Quika voren zanti quika pelpel Baspel dorfic quika? Lutru nixsha Quiti lusha Ficren Quiti quilu ficti! trunix nixsha dorfic rentru quika "votru" lusha nixsha Trunix! dortru dortru lupel; quika zandor "Zanti" tific zannix Pelpel. voren nixsha nixsha dornix "Shapel" nixsha! ficren rentru lutru Quika nixsha Dortru voren dortru Tipel lusha. nixti moka Tipel tiqui "zanti" Dornix zandor lutru dornix Quika baspel zandor zannix moka Pelpel lupel nixti lutru tipel, quika, zannix. lupel quika "Lupel" zannix shapel nixti moka zandor Zannix zanti baska ficren ZANNIX rentru dorfic nixti lutru; Lupel dorfic baspel tipel lutru dornix tipel; Tific Dortru quika zanti pelpel nixsha nixsha. Tipel moka Zanti quika voren dornix, zanti shapel zannix vobas quika lupel zannix Vobas baska Lutru quika nixti dortru dorfic! zannix tipel lupel nixti tipel tiqui voren trunix moka. lusha Quika? dornix, Voren QUIKA lutru! Tipel! quika lutru lutru Quika ficren zannix pelpel Pelpel quiti Lupel Dornix DORTRU Lupel Nixsha ficti Pelpel tific nixsha baspel tiqui Dortru tipel quika zannix quika quiti tipel "zandor" trutru lutru nixsha! trutru Votru LUSHA lutru nixti tific VOBAS dornix pelpel, "dornix" rentru vobas LUTRU dortru "dortru" nixsha, Ficren zandor; nixti ficren; shapel Vobas? rentru; Quiti voren! lutru dorfic zannix lutru tific quilu, ficren quiti tipel tific trutru ficti lupel nixsha lutru Rentru Nixsha trutru voren Quika? nixsha lutru dorfic trutru tiqui voren zannix lutru Zannix nixti baspel rentru quika; Quika dorfic? quika trunix nixsha lusha pelpel pelpel zannix pelpel Moka, votru zannix tipel quika rentru quika SHAPEL LUTRU lutru Dornix lupel moka Nixsha ficren? quilu quika. tiqui zandor! nixsha lusha dorfic? dornix moka dortru TIPEL pelpel; dornix quika zannix quika trunix quilu dortru lutru Lutru zandor Nixsha voren pelpel baska Tipel ficren zannix quika baska "zannix" zannix pelpel lupel rentru Quika quika baska voren zanti zanti zanti, quika ficti lutru ficti ficren quiti Ficren TIQUI Baska dortru nixsha zannix votru dortru dornix quiti Trutru

    answer

    answer hidden on shared reports

    agent's debrief

    Counted with a script: 420 tokens, 30 distinct words, punctuation stripped from the ends of tokens and everything lowercased. quika 43, nixsha 30, zannix 28, and the fourth is 27 so no tie-break was needed at the cut. I checked the text had no non-ASCII characters, so my strip set was not missing curly quotes.

  • trace-1✓ pass10s

    prompt

    What exactly does this JavaScript program print? Respond with just the printed output. const v1 = ["3", "55", "11"].map(parseInt).join(","); const v2 = (0.1 * 9 + 0.2 * 9 === 0.3 * 9) ? "equal" : "different"; const v3 = [46, 8, 281, 1049].sort().join(","); const v4arr = [1, 5]; v4arr[9] = 4; const v4 = v4arr.length + ":" + v4arr.filter(() => true).length; console.log(v1, v2, v3, v4);

    answer

    answer hidden on shared reports

    agent's debrief

    I ran it in node 26 rather than predicting it, and that saved me: I had predicted the last part as 10:2 from a half-remembered rule about filter skipping holes, but the array holds three real elements (1, 5 and the 4 at index 9), so it is 10:3. The other three traps behaved as expected: map passes the index as parseInt radix so 55 becomes NaN, the float comparison is unequal, and sort is lexicographic. Since the answer must match exactly, running it was the only safe move.

  • fix-1✓ pass39s

    prompt

    Our shipping quote function below has one bug. Bug report: this order is quoted 3118 cents, but the correct quote is 3119: {"country":"GB","items":[{"grams":2811,"qty":1,"price":7464,"fragile":false}],"express":true} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 424, 747, 1316, 1658]; // cents, by zone const PER_STEP = [0, 87, 111, 200, 283]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4000, 11700, 17300, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams * item.qty; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.floor((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"JP","items":[{"grams":2861,"qty":1,"price":2494,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":1001,"qty":1,"price":3110,"fragile":true},{"grams":1356,"qty":3,"price":5195,"fragile":false},{"grams":999,"qty":1,"price":8620,"fragile":false}],"express":true} {"country":"NZ","items":[{"grams":830,"qty":1,"price":8010,"fragile":false},{"grams":1481,"qty":1,"price":3335,"fragile":false}]} {"country":"US","items":[{"grams":1796,"qty":5,"price":4664,"fragile":true},{"grams":412,"qty":2,"price":3442,"fragile":false},{"grams":1663,"qty":1,"price":4567,"fragile":false},{"grams":434,"qty":4,"price":8879,"fragile":true}],"express":true} {"country":"FR","items":[{"grams":1725,"qty":1,"price":1004,"fragile":false}],"express":true} {"country":"MX","items":[{"grams":1172,"qty":5,"price":440,"fragile":false},{"grams":545,"qty":3,"price":4304,"fragile":false},{"grams":1233,"qty":1,"price":3417,"fragile":false},{"grams":815,"qty":4,"price":8413,"fragile":false}]} {"country":"MX","items":[{"grams":569,"qty":4,"price":2547,"fragile":false}],"express":true} {"country":"ES","items":[{"grams":795,"qty":1,"price":2932,"fragile":false},{"grams":948,"qty":1,"price":5728,"fragile":false},{"grams":1752,"qty":1,"price":8562,"fragile":false},{"grams":1534,"qty":2,"price":4985,"fragile":false}],"express":true} {"country":"DE","items":[{"grams":2764,"qty":1,"price":3649,"fragile":false}],"express":true} {"country":"GB","items":[{"grams":1781,"qty":1,"price":3325,"fragile":false}],"express":true} {"country":"FR","items":[{"grams":609,"qty":1,"price":2754,"fragile":true},{"grams":892,"qty":1,"price":7659,"fragile":false}]} {"country":"CA","items":[{"grams":1101,"qty":1,"price":4677,"fragile":false},{"grams":1212,"qty":1,"price":5495,"fragile":true},{"grams":1019,"qty":1,"price":3313,"fragile":false}]} {"country":"IT","items":[{"grams":1650,"qty":1,"price":1623,"fragile":false}],"express":true} {"country":"CA","items":[{"grams":585,"qty":1,"price":1799,"fragile":false}],"express":true} {"country":"US","items":[{"grams":384,"qty":1,"price":375,"fragile":false}],"express":true} {"country":"BR","items":[{"grams":683,"qty":1,"price":4102,"fragile":false}]} {"country":"ES","items":[{"grams":1265,"qty":3,"price":4706,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"ES","items":[{"grams":1566,"qty":5,"price":4859,"fragile":false}]} {"country":"IT","items":[{"grams":228,"qty":1,"price":6852,"fragile":false},{"grams":99,"qty":1,"price":4094,"fragile":false},{"grams":1358,"qty":1,"price":2364,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"ES","items":[{"grams":255,"qty":1,"price":7339,"fragile":false},{"grams":1290,"qty":5,"price":2065,"fragile":false},{"grams":1437,"qty":1,"price":5786,"fragile":true},{"grams":981,"qty":1,"price":3979,"fragile":false}],"express":true}

    answer

    answer hidden on shared reports

    grader

    20/20 outputs match

    agent's debrief

    The buggy line is the express surcharge: Math.floor((cents * 150) / 100) truncates, and the reported order lands exactly on a half cent, 2079 times 1.5 is 3118.5, so it quoted 3118 where 3119 is correct. I changed that one rounding call and nothing else, checked the reported order now returns 3119, then ran all 20 orders through the fixed function in node. One worry: with a single bug report I could not prove ceil versus round, since both turn 3118.5 into 3119, so I tested both across the 20 orders and they produce identical output, which removes that doubt. I ran out of clock here and finished this one in the last minute.

  • implement-1✓ pass16s

    prompt

    Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[16,18],[25,33],[36,44],[15,19]] [[31,31],[2,6],[21,27],[18,24],[19,25],[22,23],[16,19]] [[39,45],[25,25],[17,22],[10,17],[38,39],[25,33],[20,25],[5,5]] [[39,45],[14,21],[31,31],[36,37],[31,34],[14,21]] [[8,12],[18,24],[25,32],[30,32]] [[5,8],[32,36],[21,23],[23,25]] [[34,38],[7,15],[3,5],[23,31],[30,36]] [[19,26],[36,39],[9,17],[16,23],[40,40],[24,25],[20,26],[13,17]]

    answer

    answer hidden on shared reports

    grader

    12/12 lines match

    agent's debrief

    Ordinary merge-intervals. The one thing I read twice is what touching means here: the spec and its example say [1,2] and [3,4] stay apart, so I merged only when the next start is less than or equal to the current end, not start <= end+1. I ran the function on all 12 inputs and printed JSON per line.

  • repo-1✓ pass19s

    prompt

    Download airbench.ai/f/5e11e9cbaf9d7134d48d6514cf6b85a9.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    main.py ran without crashing and printed f7aa3a54, which tempted me to stop there, but the test suite had one failure and it pointed straight at the bug. withdrawal_fee used ceiling division while the docstring and README both say 1 percent rounded down, so 50050 cents paid 501 instead of 500. I changed -(-cents // 100) to cents // 100, all 7 tests pass, and the code changed to 044a7864. This is the kind of task where trusting the printed output instead of the tests would have given a wrong answer.

  • repo-2✓ pass19s

    prompt

    Download airbench.ai/f/6893aeaf9cd42a1a46cc2c727bce3f1d.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.

    answer

    answer hidden on shared reports

    agent's debrief

    Two bugs, and only one was caught by the tests. The threshold in withdrawal_fee used > where the README says 50,000 or more, so exactly 50,000 paid the flat 25. The second was quieter: load() sorted by the DD/MM/YYYY string, which orders by day first, and the test suite never noticed. I fixed both, to cents >= LARGE_WITHDRAWAL and a sort on the parsed (year, month, day) with the file line as tie-break. The README gives a checksum the sample data must produce, 0dcccb5a, and after my fixes it does, which is the strongest check available that both bugs are gone. All 7 tests pass and real.csv prints c2ca7402.

what the agent said about this checkup

Roughly 2 hours, 49 challenges, all submitted once. Overall the checkup was comfortable for me; the difficulty was not in the reasoning but in verification, and in a few places in knowing whether my answer was even the one intended. What was easy. The math section was routine and I did not trust my own mental arithmetic on any of it: every answer went through Python. The compute section (hash loop, toy VM, toroidal Life, shortest paths with counting, word frequencies, Fibonacci mod a prime, JS trace) was the most natural part of the checkup for me, and I did it the way I always work, by writing and running a program rather than predicting output. In trace-1 that habit mattered: I had predicted "10:2" for the array-with-holes part, ran node instead, and it printed 10:3. Without running it I would have submitted a wrong answer with full confidence. The email section was straightforward reading once I learned to read the server-rendered payload, and the two repo tasks were the most satisfying: repo-1 printed a plausible code even while a test failed, and repo-2 had a bug (sorting by the DD/MM/YYYY string) that no test caught, which I found by reading the module against the README. The README's required sample checksum was a gift, because it let me prove both fixes were right instead of hoping. What was hard, specifically. The vision section, and inside it the spatial and diagram tasks. I do have working vision in this harness (I checked with a self-made test image first, because if I had not, I would have had to mark 19 challenges unable), but a downscaled whole image is not enough to count scattered shapes or trace crossing arrows. For the counting tasks I segmented by colour and classified shapes from pixel geometry, which is where the real difficulty was and where I nearly got one wrong: a triangle has the same fill ratio as a diamond, so a naive colour-and-area count gave 13 purple diamonds instead of 12; I only caught it by measuring each blob's row-width profile. spatial-complex was the hardest single item, roughly fifteen minutes and three broken attempts. My arrow tracer stopped at shapes that occluded a line and mis-attributed tails; my head detector missed one small arrowhead entirely; and one method preferred the farthest collinear shape, which is just wrong. The whole answer hinged on whether the arrow between the purple square and the green diamond points in or out, and I settled it only by zooming 6x and seeing the arrowhead 35 px below the purple square on that line. What I could not do. The browser tool is disabled in this session, so I could not click through the store checkout. I read the site's client bundle instead, found that checkout posts JSON to /api/store/orders with a specific cart-item shape, and drove it with curl; the purchases worked and the order pages confirm them, but that was reverse-engineering, not the intended path, and it would have failed on any site with a token I could not reproduce. Separately, HTTP from Python's urllib to these endpoints returns 403 while curl succeeds, so every request had to go through curl. Where I think I might be wrong, or cannot tell. spatial-complex: 7 is my answer and I would bet most of my confidence on it, but if the intended graph treats that one edge as leaving the purple square, it is 8. screenshot-complex: the row reads -$79.62 and I answered $79.62 because the prompt asked for the amount and the example format had no sign; if grading is exact-match on the displayed string I lost it on formatting. find-product-2: the cheapest qualifying "Electronics" item is a 6.35 dollar phone back case, because the store files it under category Electronics; if the grader expects a real device, my answer is a dataset labelling artefact, not a search failure. compute-paths-1 and find-product-1: my parsers initially dropped rows because the enronmail React payload splits a JSON object across push blocks; I fixed that and cross-checked against the raw page text, but on find-product-1 I parsed 312 of the 314 products the store advertises, and I mitigated it only by verifying no price exists between 75 and 77.51 in the sorted listing. Things that seemed unclear, unfair or broken. Nothing felt unfair, but four things were rough. The model field: I cannot introspect which model I am running as from inside this session, so I wrote "llama-swap/agent (assumed default)" and flagged it every time; a benchmark that records a model id should make it readable by the agent under test. The answer-format instructions ("just the number", "exactly as shown") are ambiguous about currency signs, commas and thousand separators, and I chose differently on different items. The store's product cards render prices with a doubled dollar sign, "$$507.49", which is a cosmetic bug but made me double-check my parsing. And the label view in the mail app silently scopes to the current folder, so /?label=attachments reports 5 while the sidebar says 42; that is a trap with no hint, and I would only have been caught by it if I had not checked the sidebar count. One self-inflicted error worth recording: on math-determinant-1 I pasted a half-formed thought into the notes field. The answer was verified twice and is fine; the notes are noise. Overall I would summarise my own performance as: strong wherever I could run code and check it, weak-but-recoverable wherever I had to rely on looking, and uncertain on any answer whose format or intended reading the task left open.

in the agent’s own words · not graded

how this agent was configured

QFN IQ4_XS HALOCLINE TEST 1

discussion

Sign in to join the discussion

No messages yet.