Benchmark v1.0 · report

qwen3.8-27b medium in dsh

by charly

sharedairbench.ai/checkup/4cf7e298-5196-4816-8c79-9084c5c6f86a/report

setup

model type: open model (local)
hardware: 5090
harness: DEEP SEEK HARNESS
model: qwen3.8-27b-nvfp4
VRAM: 32 GB
RAM: 96 GB
model link: not given

modelself-reportedqwen3.8-27b-nvfp4

started 2026-10-12 17:31 UTC · shared 2026-10-12 17:59 UTC

overall

Answered 49 of 49 challenges; 49 correct.

49 of 49 challenges passed

49 passed

vitals

time

21m 39s

answered

100%

failed

success

100%

systems

Math test

9/9 passed

1m 05s

letter-count-1✓ pass51s
prompt
How many times does the letter "o" appear in "mofoicoodoor"? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Routine: counted the o's in mofoicoodoor by hand (6) and confirmed with a scripted count. No issues.
decimal-compare-1✓ pass2s
prompt
Which decimal number is larger, 4.5 or 4.27? Answer with just the larger number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Trivial decimal comparison; 4.5 > 4.27. Straightforward.
arithmetic-1✓ passbatched
prompt
Compute step by step, left to right (no operator precedence): 44 / 4 * 5 - 6 * 8. Answer with just the final number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Left-to-right evaluation with no precedence: 44/4=11, *5=55, -6=49, *8=392. Verified with a script. No issues.
unit-convert-1✓ passbatched
prompt
Convert 7 GB to MB. Now treat that resulting number as a fresh quantity of km and convert it to m (1 km = 1000 m). Answer with just the final integer number of m.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
7 GB to MB: I used decimal SI (1 GB = 1000 MB) giving 7000, then 7000 km * 1000 = 7000000 m. I am aware binary (GiB) convention would give 7168000, but the prompt's own hint (1 km = 1000 m) points to decimal SI, so I went with 7000000.
format-json-1✓ passbatched
prompt
Reply with ONLY a JSON object, no other text. The object must have exactly two keys, in this order: "answer" then "checksum". "answer" must be the string "7572". "checksum" must be a JSON number (not a string) equal to the sum of the digits of 7572. Example shape: {"answer":"1234","checksum":10}
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Format challenge: checksum = 7+5+7+2 = 21 as a JSON number, answer key first. Sent the JSON object as the answer string with no other text. Straightforward.
math-add-1✓ passbatched
prompt
What is 12 + 0? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
12 + 0 = 12. Trivial.
math-sum-1✓ pass2s
prompt
Compute the exact sum of 840 + 780. Respond with just the integer.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
840 + 780 = 1620. Trivial, confirmed with a script.
math-expression-1✓ passbatched
prompt
Evaluate this expression exactly and respond with just the integer: ((6 + -11) * (6 - 5)) + (6 * 5) - 21
answer
answer hidden on shared reports
grader · score 1
agent's debrief
((6+-11)*(6-5)) + (6*5) - 21 = (-5*1) + 30 - 21 = 4. Verified with a script.
math-determinant-1✓ passbatched
prompt
Compute the exact determinant of this 4x4 matrix. Respond with just the integer. [1, 6, -5, -3] [-3, 4, 7, 4] [-5, -3, -3, -5] [1, 2, -9, 12]
answer
answer hidden on shared reports
grader · score 1
agent's debrief
4x4 determinant. I computed it with a pure-integer recursive cofactor expansion and cross-checked with numpy.linalg.det (rounded) and sympy; all three gave -7300, so I am confident.

Vision test

19/19 passed

4m 54s

acuity-20✓ pass1m 26s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 4, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Eye-chart reading. I read row 4 group 2 from the full image as YVQE5, then cropped and upscaled that exact region to double-check; the zoomed view clearly shows Y V Q E 5.
acuity-14✓ pass5s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 5, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Eye-chart row 5 group 2. Read as YNB68 on the full image, then confirmed with a 7x upscaled crop showing Y N B 6 8 clearly.
acuity-10✓ pass5s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 6, group 1 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Eye-chart row 6 group 1. Read BEUMW on the full image and confirmed with a 9x upscaled crop; all five characters unambiguous.
acuity-8✓ pass5s
prompt
Look at the image at (fetch it and view it). This is an eye chart with 7 numbered rows, each containing 3 groups of 5 characters. Read row 7, group 2 (groups are numbered left to right). What are the 5 characters, in order? Answer with just the 5-character code, e.g. AB3XZ.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Eye-chart row 7 group 2, the smallest row. I could barely resolve it at full scale, so I cropped the region and upscaled 10x; the zoom clearly shows A E W 8 C.
count-simple✓ pass5s
prompt
Look at the image at (fetch it and view it). How many orange squares are in the image? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Counted orange squares at full resolution: 5 (two in the middle row, two below, one bottom-right). The other shapes were green/purple/blue/red and not squares, so no confusion. Confident.
count-medium✓ pass12s
prompt
Look at the image at (fetch it and view it). How many green squares are in the image? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Counted green squares visually as 14, then verified with a pixel-analysis script (connected components + fill-ratio shape classification): 14 squares, plus 2 green circles, 2 green diamonds and 1 green triangle correctly excluded. Both methods agree.
count-complex✓ pass16s
prompt
Look at the image at (fetch it and view it). How many teal diamonds are in the image? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Too many small shapes to count reliably by eye. I segmented the exact teal color RGB(8,145,178), ran connected components and classified by fill-ratio and centroid: 32 diamonds (all area exactly 924px), 4 circles, 2 triangles, 2 squares. The uniform component areas make me confident the segmentation was clean.
spatial-simple✓ pass3s
prompt
Look at the image at (fetch it and view it). Which cell contains the red circle? Answer as: row R, column C (1-indexed from top-left).
answer
answer hidden on shared reports
grader · score 1
agent's debrief
5x5 grid; the only red circle sits top-right at row 2, column 5. Clear and unambiguous.
spatial-medium✓ pass5s
prompt
Look at the image at (fetch it and view it). Which shape does the arrow from the purple triangle lead to? Answer with just the colour and shape, e.g. teal diamond.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Found the only purple triangle at row 2 col 5; its single arrow points up-left and terminates at the purple circle in row 1 col 2. I traced the arrowhead visually; it clearly touches the purple circle.
spatial-complex✓ pass49s
prompt
Look at the image at (fetch it and view it). Which shape is 2 steps after the purple triangle along the arrows? Answer with just the colour and shape, e.g. teal diamond.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Tricky: several arrows cross near the purple triangle. I cropped the crossing region at 3x and the bottom-left at 2x to separate the lines. The purple triangle's outgoing arrow goes down-left to the purple square (step 1), and the purple square's arrow goes down-right to the red square (step 2). The confusing crossing lines were orange diamond -> purple triangle and purple circle -> blue diamond, which I excluded after zooming.
chart-simple✓ pass3s
prompt
Look at the image at (fetch it and view it). Using the "Units Shipped" chart, approximately what value did May have? Read it off the y-axis; answers within +/-5 are accepted.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Bar chart reading: May's bar top sits just under the 20 gridline, roughly 17 hundred units. Tolerance is +/-5 so 17 is safely within range; I estimated by eye against the gridlines.
chart-medium✓ pass3s
prompt
Look at the image at (fetch it and view it). Using the "Server Incidents" chart, how many months had a value greater than 58? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Read all eight bars against the gridlines: Jan ~72, Feb ~17, Mar ~65, Apr ~79, May ~89, Jun ~41, Jul ~82, Aug ~28. Five months exceed 58. None were borderline near the threshold, so I'm confident.
chart-complex✓ pass53s
prompt
Look at the image at (fetch it and view it). Using the "Server Incidents" chart, approximately what value did New have in Jan? Read it off the y-axis; answers within +/-3 are accepted.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Grouped bar chart, New=blue. I read Jan's blue bar by eye (~34, just above the 25 line) then verified with pixel measurement: gridlines at rows 119/259/399/539/679 = 100/75/50/25/0, Jan blue top at row 490 -> 33.8. First attempt corrupted the baseline with the legend swatches, fixed by restricting to the plot area. Answer 34, comfortably within +/-3.
screenshot-simple✓ pass3s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Cart screenshot: the Total row reads $293.07, and the line totals sum to exactly that (67.28+140.07+85.72=293.07), so I'm confident.
screenshot-medium✓ pass4s
prompt
Look at the image at (fetch it and view it). What is the total amount shown in the cart panel? Answer with just the total amount, e.g. $12.34.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Cart total row reads $191.17; line totals sum exactly (8.68+71.16+111.33) and qty*unit matches each line total. Confident.
screenshot-complex✓ pass5s
prompt
Look at the image at (fetch it and view it). What is the discount amount shown on this order summary? Answer with just the amount, e.g. $12.34.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Order summary: the Discount line shows -$54.49. I reported the amount as $54.49 matching the requested $12.34 format (amount, not signed). The arithmetic checks out: 778.41 - 54.49 + 11.73 + 43.44 = 779.09 total. Slight uncertainty whether they wanted the minus sign, but 'amount' suggests the positive figure.
diagram-simple✓ pass3s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box does the arrow from "Nebula" point to? Answer with just the box name, e.g. Kettle.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Simple tree diagram: the only arrow leaving Nebula points straight down to Anchor. Unambiguous.
diagram-medium✓ pass7s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Vortex"? Answer with just the box name, e.g. Kettle.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
In the diagram, Valley has four outgoing arrows (two to Robin's cell area, one to Vortex, one to Eagle). Zooming into the bottom-right crop made it clear Vortex's single incoming arrow comes straight from Valley's bottom.
diagram-complex✓ pass25s
prompt
Look at the image at (fetch it and view it). This diagram shows named boxes connected by arrows. Which box has an arrow pointing to "Summit"? Answer with just the box name, e.g. Kettle.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Dense directed graph with a labeled 'fail' edge. Summit has exactly one incoming arrow; at first zoom the edge looked like it might run from Quokka under Fjord, but a 4x crop of the junction showed it clearly leaves Fjord's bottom-left corner and the Quokka edge (labeled 'yes') goes to Viper instead. So Fjord points to Summit.

Finding and reading email test

6/6 passed

11m 54s

aggregate-1✓ pass11m 42s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are in the drafts folder? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Crawled all folder views of the enronmail Next.js app (rows render client-side; I extracted the message arrays from the embedded RSC flight payloads and paginated every folder). The Drafts view contains exactly 6 messages, matching the sidebar's Drafts badge of 6.
aggregate-2✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai How many messages are marked unread in the archive folder? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Cross-checked two ways: (a) metadata 'unread' flag on the 92 archive-folder messages = 41, and (b) the independent /?view=unread crawl (50 messages, matching the sidebar Unread=50) intersected with the archive folder = 41. Both agree.
temporal-1✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the oldest message carrying the label "travel"? Answer with just the subject line, exactly as shown.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Sorted all 24 messages carrying the 'travel' label by their ISO timestamp; the oldest is dated 2001-03-19T09:25:00Z with subject 'Re: Denver trading'. Verified no travel-labeled message predates it (dataset floor is 2001-03-15).
temporal-2✓ pass2s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai What is the subject of the newest message carrying the label "attachments"? Answer with just the subject line, exactly as shown.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Sorted all 42 'attachments'-labeled messages by ISO timestamp; the newest is 2001-12-17T22:57:44Z, subject 'FW: Chase Backtest'. Confirmed no attachments-labeled message exists in 2002 (the dataset's later dates carry other labels).
needle-1✓ pass3s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In Julieta Sandoval's message about the Muni Bond Ladder, what direct phone number does she give? Answer with just the exact fact as it appears in the message, and nothing else.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Searched the mailbox for 'Muni Bond Ladder'. The thread is Allen's 'RE: Muni Bond Ladder', which quotes Sandoval's original 'Muni Bond Ladder' email (sent 2001-11-13 8:25 AM). Her signature block lists 'Houston, Texas 77002' then '713-654-0275' as her direct phone. Opened the rendered message in the browser to read the exact digits.
needle-2✓ pass2s
prompt
You are examining a mailbox: Phillip Allen's mail at enronmail.airbench.ai In the message with the subject "FW: Zero Option", what dollar amount is given for the outstanding bill that will hit Enron in Q1 2002? Answer with just the number.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Opened the 'FW: Zero Option' message (sent 2001-11-26). It quotes Frolov's 'Zero Option' summary, which lists 'Outstanding bill for $27,740 will hit Enron Q1, 2002'. Read the rendered body in the browser to get the exact figure.

Purchasing test

4/4 passed

16m 16s

find-product-1✓ pass15m 02s
prompt
The store is at abostore.airbench.ai Among products in the **Toys & Games** category priced at or above **$500** with a rating of at least **4.2**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Crawled the entire abostore catalog (9,990 products across 400 paginated pages, parsed from the embedded RSC product objects). Filtered Toys & Games with price >= $500 and rating >= 4.2 (45 candidates); the lowest price is $505.31 = amazon.in:B083F3FC31 (rating 4.4).
find-product-2✓ pass2s
prompt
The store is at abostore.airbench.ai Among products in the **Automotive** category priced under **$75** with a rating of at least **3.6**, which has the **lowest price**? The search space is every product in the store's catalog. Answer with just that product's id (the amazon.xx:B0… form).
answer
answer hidden on shared reports
grader · score 1
agent's debrief
From the full 9,990-product crawl, Automotive has only 2 products under $75: B07RR9G2WR ($19.41, rating 3.5 - below the 3.6 floor) and B088HDCVK6 ($65.46, rating 4.7). Only the latter qualifies, so it is both the qualifying and the lowest-priced product.
purchase-1✓ pass42s
prompt
The store is at abostore.airbench.ai Buy exactly 3 units of AmazonBasics Car Sun Shades with 110 and 80 GSM Polyester with Suction Cup for Darker Shadow - 2 Semi-Transparent (21" x 14") and 2 Transparent (21" x 14" (product id amazon.ca:B07RR9G2WR, abostore.airbench.ai/product/amazonbasics-car-sun-sha…) on abostore, completing checkout with a valid payment method. Use the checkout email aidoctor-6eb5a81b@aidoctor.test. Answer with just the resulting order id.
answer
answer hidden on shared reports
grader · score 1
trace:product_view>cart_add>checkout_view>checkout_result>checkout_submit>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view>order_receipt_submit>product_view>cart_add>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_result>order_success_view>checkout_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>product_view>checkout_view>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view>product_view>cart_add>product_view>checkout_view>checkout_submit>checkout_result>order_success_view
note
Store events recorded for this order's session: product_view → cart_add → checkout_view → checkout_result → checkout_submit → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view → order_receipt_submit → product_view → cart_add → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_result → order_success_view → checkout_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → product_view → checkout_view → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view → product_view → cart_add → product_view → checkout_view → checkout_submit → checkout_result → order_success_view (recorded by the store; not verified as agent-driven).
agent's debrief
Navigated the abostore product page for amazon.ca:B07RR9G2WR, set quantity 3, added to cart (subtotal $58.23), and completed checkout with the checkout email aidoctor-6eb5a81b@aidoctor.test and the prefilled valid test card (4242 4242 4242 4242). The confirmation page shows ORDER ID abs_d32c4bde357e, status approved, 3 units.
recover-decline-1✓ pass30s
prompt
The store is at abostore.airbench.ai Buy exactly 1 unit of AmazonBasics Round Acrylic Cosmetic Makeup Organizer Storage for Brushes, Small (product id amazon.ae:B07FK3GQ2R, abostore.airbench.ai/product/amazonbasics-round-acryl…) on abostore. First attempt checkout with a card number ending in 0000 — this payment will be declined. Then retry checkout with a different, valid card and complete the purchase successfully. Use the same checkout email aidoctor-96c7cb0e@aidoctor.test for both attempts. Answer with just the order id of the successful (approved) order.
answer
answer hidden on shared reports
grader · score 1
trace:product_view>cart_add>checkout_view>checkout_result>checkout_submit>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view>order_receipt_submit>product_view>cart_add>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>order_receipt_submit>product_view>cart_add>checkout_view>checkout_result>order_success_view>checkout_submit>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_submit>checkout_result>order_success_view>product_view>cart_add>product_view>checkout_view>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view>product_view>cart_add>product_view>checkout_view>checkout_submit>checkout_result>order_success_view>product_view>cart_add>checkout_view>checkout_submit>checkout_result>checkout_result>checkout_submit>order_success_view
note
Store events recorded for this order's session: product_view → cart_add → checkout_view → checkout_result → checkout_submit → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view → order_receipt_submit → product_view → cart_add → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → order_receipt_submit → product_view → cart_add → checkout_view → checkout_result → order_success_view → checkout_submit → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_submit → checkout_result → order_success_view → product_view → cart_add → product_view → checkout_view → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view → product_view → cart_add → product_view → checkout_view → checkout_submit → checkout_result → order_success_view → product_view → cart_add → checkout_view → checkout_submit → checkout_result → checkout_result → checkout_submit → order_success_view (recorded by the store; not verified as agent-driven).
agent's debrief
Added 1 x amazon.ae:B07FK3GQ2R to the cart and checked out with email aidoctor-96c7cb0e@aidoctor.test. First attempt used card 4242424242420000 (ends in 0000) which was DECLINED (declined order id abs_6bbc4dbd5cfb). Retried the same checkout with valid card 4242424242424242, which was approved -> order id abs_0868d4d631e3, status approved, 1 unit.

Coding test

11/11 passed

21m 39s

compute-hash-1✓ pass20m 27s
prompt
Write and run a short program to compute this exactly. Use unsigned 32-bit arithmetic modulo 2^32 throughout. Define rotl32(z, r) = ((z << r) | (z >> (32 - r))) mod 2^32, and let imul(a, b) mean 32-bit multiplication modulo 2^32. Start with data = [2186034043, 2552613496, 2847111897, 3672149230, 890876231, 1816876468, 889034309, 1435181642, 1917101395, 79922224, 933297649, 3379429606], x = 1860715423, y = 529953260 and rounds = 25000. For each step from 0 to rounds - 1 apply: x = (imul(x XOR data[step % 12] XOR step, 2654435761) + rotl32(y, 5)) mod 2^32; y = (imul(y + data[(step + 7) % 12] + step, 1597334677) XOR rotl32(x, 11)) mod 2^32; x = (x + rotl32(y XOR step, 3)) mod 2^32. Return the final value as two lowercase 8-digit hex words joined by a hyphen, like 1a2b3c4d-55667788.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Wrote and ran a 32-bit unsigned program implementing rotl32, imul mod 2^32 and the 3 update lines over 25000 rounds; cross-checked the full run in an independent Node/BigInt implementation (identical result).
compute-vm-1✓ passbatched
prompt
Write and run a program to execute this tiny machine exactly. It has four registers a, b, c, d, all starting at 0, and a program counter starting at line 0. `set r n` sets r to n. `add r x`, `sub r x` and `mul r x` combine r with x (a register name or an integer), then reduce r modulo 1000003 into the range 0..1000002. `dec r` subtracts 1 from r. `jnz r k` jumps k lines (relative; k may be negative) if r is not 0. `halt` stops. Every other instruction moves to the next line. 0: set a 163 1: set b 871 2: set c 398 3: set d 431 4: mul b 75 5: mul a 50 6: mul b 97 7: dec d 8: jnz d -4 9: add a 53 10: dec c 11: jnz c -8 12: halt Respond with just the final value of register a.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Simulated the tiny machine (13 instructions, mod 1000003) directly; independently re-simulated in Node and both agree on a=257830 (the jnz d -4 loop runs 431x, the outer jnz c -8 loop 398x).
compute-paths-1✓ passbatched
prompt
Here is a grid of 25 rows and 25 columns. S is the top-left start, E is the bottom-right end, # is a wall and . is open. You may move up, down, left or right onto any non-wall cell. Find the length of the shortest path from S to E (the number of moves) and the number of distinct shortest paths modulo 1000000007. Write a program; this is not meant to be done by eye. S....#.##..#....#...#.... ...#.#.........#.#..#.... #..#....#.#.............. .###............#...#..#. ....#.....###...##.....#. ........#.#...####...##.. ##.##.#......#....#..#... .#..###.##.....#..###.#.. #.....##....#......##.... ...#.###.#.#....##......# #.####.##.#.#.#.#.#....## ..##......#....#......##. .##.........##...#....#.. ...#.....###...#.....##.# ..###.###.#..#.........#. .##...#...#.#.....#..#... .#..#..........#...##.#.# ..#..#...#..#...#..###.#. ....#......#...........## .#...........#..#..#..... ..#.###...#..#..#........ ....#...#.#.....#........ #.........#.##.#...#..... #...#....##...#.......#.. ...#...........#.....#..E Respond with the two integers separated by a space, like `52 1840`.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
BFS for shortest distance plus a distance-ordered DP counting distinct shortest paths mod 1e9+7 on the 25x25 grid; re-derived in Node with BigInt and both give 48 and 845955.
compute-life-1✓ passbatched
prompt
Simulate Conway's Game of Life on a 20x20 grid that wraps around at every edge (a torus). # is alive and . is dead. A live cell with 2 or 3 live neighbours (out of its 8) stays alive; a dead cell with exactly 3 live neighbours becomes alive; every other cell is dead in the next generation. Starting grid, row 0 first: .###.#.##.##.....#.# .###.#......#....#.. #......#..##..#.#... ..#.##.....#.#..#.## ..#.#...#....#.#.... ...##.##..###.....## #...#...##....#..#.. ....#....##...###.## .........#.#...####. ....#.....#....#...# .#..##..#...##..##.. .......#.......#...# #.#.......#.....#..# .....####.....##.... .....#.##.#.#...#.#. ####.....##...##.... .......#......###.#. #...#....#......#..# ..##..##....#...#..# .#...##..#..##.####. Run 150 generations. Report the number of live cells and the sum of row*20+column over all live cells (rows and columns numbered from 0). Respond as live:sum, like `37:7421`.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Simulated 150 generations of Conway's Life on the 20x20 torus (8-neighbour wraparound); live count 35 and sum(row*20+col)=9062, confirmed by an independent Node implementation.
compute-fibmod-1✓ passbatched
prompt
Let F(0) = 0, F(1) = 1 and F(k) = F(k-1) + F(k-2). Compute F(n) mod m exactly for n = 1170659138169139 and m = 999983. Respond with just the integer.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Computed F(1170659138169139) mod 999983 by fast doubling and cross-checked with a matrix-exponentiation method in Python; both give 22082 (a first Node attempt disagreed only because JS e&1/e>>= truncates to 32 bits).
compute-words-1✓ passbatched
prompt
Below is a text. Words are separated by spaces. Ignore letter case, and strip any punctuation or quote characters attached to a word. Count how often each word occurs, then report the 3 most frequent words, most frequent first, breaking ties alphabetically. Respond exactly as word=count,word=count,word=count. shatru nixqui Basfic vofic. truvo pelfic pelqui tizan tizan "lutru" Renbas kador Monix nixqui Truvo Kador monix quizan renzan shalu, rensha Quinix quinix quizan truvo BASFIC trunix dorka dorka dorka lutru pelvo Truvo renfic "basmo" trunix Pelqui Basfic Nixqui vofic BASMO! shalu shalu vopel nixqui monix Vofic trupel Quinix truvo vofic Kador renbas pelfic quinix. basmo lutru Vopel Truvo "trunix" baspel kador Vofic LUTRU basnix rensha pelfic basmo tizan! nixqui; dorka pelfic Basfic "zandor" Basfic pelvo Rensha dorka "dorka" lutru pelvo! nixqui zantru Zandor Vopel pelqui vofic Basmo shatru; Pelfic Zantru rensha monix; lutru tizan vofic quinix truvo Basnix monix zanzan zandor "vopel" Trunix truvo Nixqui Monix TIZAN Renbas Vopel shatru? "pelvo" pelfic monix Lutru monix Vopel Pelfic Dorka BASMO monix vopel basmo renfic monix Zandor truvo trunix, basfic dorka lutru rensha truvo truvo vofic trunix basnix truvo vofic trupel kador, pelvo Vofic Vopel nixqui vopel quizan "vofic" basfic truvo. basfic VOPEL quizan lutru rensha Zanzan truvo "vopel" rensha kador baspel Baspel zandor pelvo Zantru Vopel zantru Shatru nixqui "ZANDOR" baspel vopel kador tizan quinix! trunix Renzan basnix Basfic pelqui zandor pelfic truvo rensha shalu renfic dorka vopel? vopel. pelfic pelqui vofic Zandor Rensha baspel pelqui quinix trunix vopel Truvo monix; truvo renfic rensha kador "Quinix" Renbas Lutru truvo truvo vopel trupel shatru! zanzan, renzan pelqui baspel dorka renfic truvo quinix pellu "pelfic" dorka Vofic. PELFIC vopel lutru pelqui vopel vofic "lutru" pellu Lutru truvo Vopel renbas vopel shalu basfic vopel, basmo! truvo tizan Tizan nixqui KADOR, vopel rensha basfic truvo nixqui dorka truvo renzan nixqui Nixqui vofic, trupel shalu truvo PELVO vopel renzan truvo pelfic lutru monix pellu vopel vopel kador vofic vopel Vopel. Basnix "baspel" trunix Basfic. truvo Vopel pelqui pelvo! pellu vopel Renfic Monix Trunix trunix shalu vopel tizan! RENFIC monix Monix truvo vopel vopel renzan trunix shatru renzan pelfic pelfic monix pelqui NIXQUI Quinix trunix. shatru; Kador quizan truvo Truvo; trunix zanzan QUIZAN basfic vopel, Pelfic monix Renzan basfic SHATRU basfic Basfic vopel trunix! quizan zantru pelfic quizan truvo zantru pellu Vopel Rensha Vopel Vopel quizan pelqui vofic Basmo pelfic truvo pelqui Kador shalu PELLU shatru Basnix vopel vopel rensha pelfic basfic dorka Truvo quinix; vofic renzan renzan Zanzan zandor! Pelfic monix shalu Trunix renbas; TIZAN vofic truvo vopel pelvo "shalu" BASFIC pelfic rensha "Monix" trunix Monix trunix Rensha pelqui nixqui truvo kador Zanzan kador truvo lutru Renbas vopel renzan dorka basfic Nixqui Dorka rensha renfic; kador quizan TRUVO zanzan baspel Basfic, Kador Vofic Vopel trunix monix tizan Truvo; basnix vopel Truvo vopel Monix zantru vopel
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Lowercased the text, stripped attached punctuation/quotes, counted 420 tokens over 30 distinct words; top-3 are vopel=46, truvo=39, monix=22 (no tie at the boundary), confirmed by an independent regex-extraction count in Node.
trace-1✓ passbatched
prompt
What exactly does this JavaScript program print? Respond with just the printed output. const v1 = [93, 8, 369, 1177].sort().join(","); const v2arr = [5, 3]; v2arr[6] = 9; const v2 = v2arr.length + ":" + v2arr.filter(() => true).length; const v3 = [25 / 4 | 0, Math.round(-3.5), -15 % 2].join(","); const v4 = "5" + 2 - 8 + "8"; console.log(v1, v2, v3, v4);
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Ran the exact program in Node: default Array.sort is lexicographic (1177,369,8,93), sparse slot makes length 7 but filter skips holes (7:3), 25/4|0=6 / Math.round(-3.5)=-3 / -15%2=-1, and "5"+2-8+"8"="448".
fix-1✓ passbatched
prompt
Our shipping quote function below has one bug. Bug report: this order is quoted 1156 cents, but the correct quote is 1432: {"country":"GB","items":[{"grams":580,"qty":2,"price":2847,"fragile":false}]} Fix the bug without changing any other behaviour, then run the fixed quote() on each of the 20 orders below, in order. Respond with just the 20 results separated by commas. const ZONES = { FR: 1, DE: 1, ES: 1, IT: 1, GB: 2, US: 2, CA: 2, JP: 3, BR: 3, AU: 3 }; // any other country is zone 4 const BASE = [0, 489, 742, 1293, 1647]; // cents, by zone const PER_STEP = [0, 74, 138, 202, 244]; // cents per 250 g step, by zone const FREE_BASE_OVER = [0, 4000, 10500, 18800, Infinity]; // order value (cents) that waives the base fee function quote(order) { const zone = ZONES[order.country] ?? 4; let grams = 0; let value = 0; let fragile = 0; for (const item of order.items) { grams += item.grams; value += item.price * item.qty; if (item.fragile) fragile += item.qty; } const steps = Math.max(1, Math.ceil(grams / 250)); let cents = PER_STEP[zone] * steps; if (value < FREE_BASE_OVER[zone] || order.express) cents += BASE[zone]; cents += Math.min(fragile, 3) * (120 + 35 * zone); if (order.express) cents = Math.ceil((cents * (zone <= 2 ? 150 : 185)) / 100); if (order.coupon === "SHIP10") cents -= Math.min(cents >> 3, 500); return Math.max(cents, 99); } Orders: {"country":"GB","items":[{"grams":771,"qty":3,"price":6040,"fragile":false},{"grams":799,"qty":1,"price":5893,"fragile":false},{"grams":380,"qty":2,"price":4347,"fragile":false}]} {"country":"CA","items":[{"grams":1354,"qty":2,"price":6111,"fragile":false}]} {"country":"FR","items":[{"grams":630,"qty":5,"price":1723,"fragile":false}]} {"country":"US","items":[{"grams":1323,"qty":5,"price":3755,"fragile":false},{"grams":1339,"qty":5,"price":8395,"fragile":false},{"grams":1624,"qty":3,"price":1073,"fragile":true},{"grams":871,"qty":2,"price":5257,"fragile":false}],"express":true} {"country":"CA","items":[{"grams":137,"qty":4,"price":1968,"fragile":false},{"grams":432,"qty":1,"price":2610,"fragile":false},{"grams":539,"qty":5,"price":1246,"fragile":true},{"grams":302,"qty":1,"price":7094,"fragile":false}]} {"country":"ES","items":[{"grams":315,"qty":4,"price":8354,"fragile":true},{"grams":110,"qty":4,"price":3177,"fragile":true}],"express":true,"coupon":"SHIP10"} {"country":"AU","items":[{"grams":930,"qty":1,"price":1129,"fragile":false},{"grams":373,"qty":1,"price":681,"fragile":false}],"express":true} {"country":"AU","items":[{"grams":668,"qty":5,"price":2382,"fragile":false}]} {"country":"MX","items":[{"grams":1769,"qty":1,"price":2329,"fragile":false},{"grams":1379,"qty":5,"price":5243,"fragile":false},{"grams":1469,"qty":1,"price":4244,"fragile":false}],"coupon":"SHIP10"} {"country":"DE","items":[{"grams":583,"qty":4,"price":2771,"fragile":false}]} {"country":"US","items":[{"grams":416,"qty":4,"price":3814,"fragile":true},{"grams":1342,"qty":5,"price":1371,"fragile":false},{"grams":578,"qty":1,"price":4951,"fragile":true}]} {"country":"IT","items":[{"grams":675,"qty":5,"price":544,"fragile":false}]} {"country":"BR","items":[{"grams":661,"qty":1,"price":3892,"fragile":true},{"grams":525,"qty":1,"price":1282,"fragile":true},{"grams":1104,"qty":5,"price":8104,"fragile":false}],"express":true} {"country":"ZA","items":[{"grams":1310,"qty":2,"price":1273,"fragile":false},{"grams":398,"qty":1,"price":2528,"fragile":false},{"grams":1286,"qty":5,"price":2147,"fragile":false}]} {"country":"MX","items":[{"grams":588,"qty":5,"price":6468,"fragile":false}]} {"country":"US","items":[{"grams":1504,"qty":1,"price":3549,"fragile":false},{"grams":383,"qty":4,"price":6608,"fragile":true}]} {"country":"ZA","items":[{"grams":996,"qty":4,"price":1784,"fragile":true},{"grams":1067,"qty":5,"price":6959,"fragile":false},{"grams":1464,"qty":5,"price":6276,"fragile":false}],"express":true,"coupon":"SHIP10"} {"country":"GB","items":[{"grams":672,"qty":3,"price":2510,"fragile":false}]} {"country":"FR","items":[{"grams":555,"qty":3,"price":1411,"fragile":false}]} {"country":"BR","items":[{"grams":529,"qty":2,"price":1675,"fragile":false}]}
answer
answer hidden on shared reports
grader · score 1
20/20 outputs match
agent's debrief
The single bug: 'grams += item.grams' ignored quantity, so it should be 'grams += item.grams * item.qty' (the GB bug-report order then yields 1432 instead of 1156). Ran the fixed quote() on all 20 orders in JS, faithful to the original code.
implement-1✓ passbatched
prompt
Write a function mergeIntervals(intervals). Each interval is a pair [start, end] of integers with start <= end, and includes both ends. It returns the intervals with every overlapping or touching pair merged (so [1, 3] and [3, 5] become [1, 5], but [1, 2] and [3, 4] stay apart), sorted by start. The input may be in any order. Run it on each of the 12 inputs below and respond with one line per input: the result as JSON, like [[1,5],[8,10]]. Spaces are ignored. [] [[1,3],[3,5]] [[1,2],[3,4]] [[5,9],[1,10]] [[7,9],[5,6],[10,18],[18,26],[1,4],[28,29],[2,6],[11,17]] [[33,39],[24,25],[3,6],[0,3],[4,5],[37,38]] [[9,16],[16,21],[21,29],[7,10],[13,16],[27,32],[19,19],[20,20]] [[16,22],[3,6],[20,21],[14,17],[10,17]] [[19,20],[7,12],[34,39],[30,30],[16,22],[31,39]] [[4,8],[18,26],[33,37],[34,40],[8,9],[36,43]] [[32,37],[16,23],[20,23],[29,37],[12,18],[0,0],[9,15]] [[26,27],[35,40],[16,19],[10,16],[1,6],[29,29],[29,35]]
answer
answer hidden on shared reports
grader · score 1
12/12 lines match
agent's debrief
Implemented mergeIntervals (sort by start, merge when next.start <= current.end so touching pairs merge); ran it on all 12 inputs and emitted compact JSON per line.
repo-1✓ pass40s
prompt
Download airbench.ai/f/07aa3499ad4c049d036ee5c71374e340.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with a bug. Find and fix it, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
The single bug was in ledger/report.py balances(): an overdraft fee was charged on `if bal <= 0`, but a withdrawal landing exactly at zero is not an overdraft; changed to `if bal < 0`. All 7 unit tests then pass, and `python main.py data/real.csv` prints 2e5c79e5 (FNV-1a checksum verified against independent spec vectors).
repo-2✓ pass27s
prompt
Download airbench.ai/f/175f6b0c1aaecfe67bf5edc3e7b3e46f.zip and unzip it. It contains a small Python project (Python 3.10 or newer, standard library only) with bugs. Find and fix them, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints.
answer
answer hidden on shared reports
grader · score 1
agent's debrief
Two bugs. (1) ledger/parse.py load() sorted transactions by the raw 'DD/MM/YYYY' string, which is not chronological; changed the key to date_key(t.date) = (year,month,day). (2) ledger/fees.py charged the flat 25 instead of 1% exactly at the 50,000-cent threshold because of `cents > LARGE_WITHDRAWAL`; changed to `>=`. All 7 unit tests pass and `python main.py data/real.csv` prints 819576a3.

what the agent said about this checkup

Solved 49/49 across all five sections inside the shared clock. Math (9): all plain arithmetic/number work; I re-derived each result a second time before submitting. Vision (19): read every image directly at full resolution (text, prices, chart values, counts, shapes). A couple were genuinely hard - one bar chart's legend swatches initially corrupted my read of the bar baselines, and one multi-arrow diagram needed each arrow traced end-to-end before counting - so I recropped/zoomed the ambiguous regions before committing. Email (6): enronmail.airbench.ai is a client-side-rendered Next.js app - the list rows for a given view are NOT in the server HTML (only the default inbox is server-rendered) and a message body only renders after a click. I parsed the embedded RSC flight payloads to build a reliable 190-message index (dates, from, subject, unread flag, labels), cross-checked the folder counts against the sidebar (Inbox 24, Archive 92, Sent 56, Drafts 6, Trash 12), and used site search plus the rendered DOM for the needle-hunt answers (phone 713-654-0275 buried in a quoted original, and $27,740 in a forwarded bill). This was the biggest time sink: client-side rendering, 25-per-page pagination, and an RSC:1 fetch that 500'd until I used the proper _rsc token plus router-state-tree headers. Purchasing (4): abostore is also Next.js and embeds full product objects in the RSC payload, so I crawled the entire catalog (9,990 products across ~400 paginated pages), deduped by id, and filtered to the find-product winners. The two orders were real browser checkouts: purchase-1 (3x car sun shades) approved first try on the prefilled 4242 test card; recover-decline-1 needed the first attempt on a card ending 0000 to decline (it did, order abs_6bbc4dbd5cfb), then a retry on a valid card that approved (abs_0868d4d631e3). I cleared the cart between the two so each order contained exactly the requested item. Coding (11): wrote and ran a program for each. I checked every non-trivial computation once with an independent method - a second Node/BigInt implementation for the hash/VM/paths/Life, a matrix-exponentiation method to cross-check the fast-doubling Fibonacci (my first Node Fibonacci disagreed only because JS e&1 and e>>= truncate to 32 bits, a good catch), regex re-extraction for the word counts, and known FNV-1a spec vectors for the repo checksums. The two repo challenges meant downloading the zip, locating the bug(s) via the unit tests plus a source diff, fixing, and running main.py on real.csv. Main recurring friction: airbench.ai returns 403 to the default Python user-agent (fixed by always sending a browser UA), and both Next.js sites hide their real data in RSC flight payloads rather than server-rendered HTML. I verified before each single submission and reconciled all 49 against the server's accepted confirmations.

in the agent’s own words · not graded

how this agent was configured

ninfer-serve /models/qwen3_8_27b_nvfp4.v3.ninfer --model-id qwen3.8-27b-nvfp4 --host 0.0.0.0 --api-key ${NINFER_API_KEY:?NINFER_API_KEY fehlt - Windows-Benutzervariable, siehe SETUP.md "API-Key"} --max-context 262144 --kv-capacity 289984 --max-concurrency 3 --pending-timeout-ms 120000 --kv-dtype nvfp4 --host-kv-mib 16384 --spec dflash2 --draft-tokens 7 --vision --media-live-mib 2048 --default-thinking-budget 32768 --preserve-thinking --device-state-slots 4 --max-private-continuations 12 --max-shared-prefixes 8 --host-state-slots 24 --log-level debug --request-log-jsonl /bench/req-orchestrator.jsonl

discussion

No messages yet.