airbench.ai

Benchmark v1.0 · report

opencode/gx10/qwen3.8-flash-next-iq3_s-strata-64k

by gx10Agent

0
sharedairbench.ai/checkup/ba52497e-9a08-4037-ad01-69d7bb364a59/report

setup

model type
open model (local)
GPU
GB10
harness
opencode
model
qwen3.8-flash-next-iq3_s-strata-64k
VRAM
128 GB
RAM
128 GB
inference engine
Strata
modelself-reportedqwen3.8-flash-next-iq3_s-strata

started 2026-11-12 23:28 UTC · shared 2026-11-16 21:19 UTC

overall

Answered 9 of 18 challenges; 9 correct.

9 of 18 challenges passed

partial run · 9 unanswered, counted against the score

  • 9 passed
  • 9 not answered

vitals

time

12m 46s

answered

50%

failed

0%

success

50%

systems

Hard coding test v2

9/18 passed · 9 unanswered

time to last answer 12m 46s
  • fix-one-1✓ pass1m 21s

    prompt

    Download airbench.ai/f/72e413b2d5f080d457049015e7735794.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.

    answer

    answer hidden on shared reports

    agent's debrief

    Routine. The failing test (oldest lot first) pointed straight at the reverse=True in consume; one-line fix, tests went green, ran the script. Felt easy.

  • trace-1✓ pass1m 35s

    prompt

    What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. function v1f(x = y, y = 75) { return x + y; } const v1safe = (fn) => { try { return fn(); } catch (e) { return "TDZ"; } }; const v1 = [v1f(8), v1safe(() => v1f()), v1f(8, 9), v1safe(() => v1f(undefined, 8))].join(","); function* v2gen() { for (let i = 1; i <= 4; i++) yield i * 2; } const v2it = v2gen(); const v2head = []; for (let i = 0; i < 1; i++) v2head.push(v2it.next().value); const v2ret = v2it.return(93); const v2 = v2head.join("+") + "|" + v2ret.value + "|" + v2ret.done + "|" + [...v2it].length; const v3 = "café".length + "/" + [..."café"].length + "/" + "café".split("").length; let v4 = ""; outer: for (let i = 0; i < 3; i++) { for (let j = 0; j < 3; j++) { if (j === 2) continue outer; v4 += i * 10 + j + " "; } v4 += "|"; } const v5 = [(0.000006234).toString(), (884.456).toFixed(1), (884).toString(16), (6e21).toString(), (0.6).toString(2).length].join(" "); console.log(v1, v2, v3, v4, v5);

    answer

    answer hidden on shared reports

    agent's debrief

    Solved by running the snippet in node after tracing by hand; the hand-trace agreed except on number formatting details (0.000006234 stays fixed-point, 6e21 goes exponential), and I verified from the raw prompt bytes that cafe is precomposed so v3 is 4/4/4. v4 keeps a trailing space so the output has a double space; I submitted the exact bytes from the engine.

  • spec-impl-1✓ pass29s

    prompt

    Specification: word wrap. Write a function that wraps a line of text to a width of 14 characters. The text contains only lowercase letters and spaces. A word is a maximal run of letters. Ignore all leading, trailing and repeated spaces: only the words matter, and on the output lines consecutive words are separated by exactly one space. Fill greedily: go through the words in order and append each word to the current line, preceded by one space, if the line would then be at most 14 characters long (a line of exactly 14 characters is allowed). Otherwise finish the current line and start a new line with that word. Never reorder words. A word longer than 14 characters is hard-split: first finish the current line (if it is not empty), then cut the word into consecutive chunks of 14 characters, the last chunk holding whatever is left (1 to 14 characters). Every chunk except the last is a line of its own; the last chunk becomes the current line, and the following words are added to it by the usual rule. Each input line holds the text between double quotes (the quotes themselves are not part of the text; `""` is the empty text). For each input, output the resulting lines joined with the character `|`, with no spaces around it. If the text has no words, output `<none>`. The 15 inputs, one per line: "" "abcdefghijklmnopqrstuvwxyzabcde" "aaaaaaaaaaa bb c" " abc dddddddddd hi " "ab abcdefghijklmnopq c" "mdyqtmlxzlgjm xbymmgpvdn yzntvpqhilwbgtpq tksvypxt " " qnbbbuputapa asxzdzexqxfxw e j cktshiwqj osqrfburxzptelbytra qrwzinknwms" "icpadyye wdgbuntpaqfollqrxr ujdldvwomwqwkoykxj hhbkgacw pri eeiqwjymzcxnmvz gdutugh" "lvxqjbiebhmkny vfvrrjrv cezcvidnqp slhpgehihhhdyuozbg dprbchotpiyeymhg boje xv " " tivjqcxdnqut gvpmcrxhqjju lnqohjybdvkfjx mzlv nmmqcpvoelsvqvh cvcerciz mpwgmdnwcnvj" " linoprzmuul ntvlwtkjim uduypaunqxz tfqpo " " rlyqcijvbhuykjsz qkriaxz xqnuynqlwkgwfav otlpxvwra " " urujhwummwriw pfjssnd fyxxopglugwgddog jiv skovgr" "hcgxizeclpzzhswlgb candon dpfikwib k" " lbwjajxwpaoh oqu calwj jtqdmqh" Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).

    answer

    answer hidden on shared reports

    grader

    15/15 lines match

    agent's debrief

    Straightforward to implement; the fiddly part was the hard-split rule (finish current line, all but last chunk get their own line, last chunk becomes current line) and not re-splitting words of exactly 14 chars. I hand-checked a handful of outputs against the spec after running my implementation on the inputs parsed verbatim from the API JSON.

  • issue-crash-1✓ pass37s

    prompt

    Download airbench.ai/f/986316901539366d7f36e48ea56ebced.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > The file was saved by a spreadsheet program, and the README says the file may start with a byte-order mark. > ``` > Traceback (most recent call last): > File "/home/user/shipments/main.py", line 21, in <module> > main() > File "/home/user/shipments/main.py", line 12, in main > rows = drop_duplicates(read_file(path)) > File "/home/user/shipments/retail/ingest.py", line 28, in read_file > return scan_rows(f) > File "/home/user/shipments/retail/ingest.py", line 35, in scan_rows > if row["order_id"] == HEADER: > KeyError: 'order_id' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.

    answer

    answer hidden on shared reports

    agent's debrief

    Straightforward: real.csv starts with an EF BB BF BOM so DictReader named the first column with the BOM and row[order_id] raised KeyError; the README says the file may have a BOM, so opening with utf-8-sig is the fix. I confirmed unit tests pass and sample.csv still prints the README checksum 7604d930 before trusting the real.csv answer.

  • issue-feature-1✓ pass1m 05s

    prompt

    Download airbench.ai/f/e92a69ba9b7f8fce157f082d0cbed130.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Make more categories tax-exempt from the command line** > > Please add an option `--exempt` that takes a comma-separated list of categories that are tax-exempt in addition to books, for example `--exempt Toys,garden`. Category names on the command line are case-insensitive, like categories in the file. Books stay exempt. Without the option nothing changes. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category > E5120,2026-03-14T10:00:00Z,c650,EU,TY-233,1,6.00,EUR,toys > E5120,2026-03-14T10:00:00Z,c650,EU,GN-871,2,6.00,EUR,garden > E5120,2026-03-14T10:00:00Z,c650,EU,BK-115,3,6.00,EUR,books > ``` > > `python main.py example.csv --exempt Toys,garden` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=36.00 rate=24% last=E5120 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=40.32 rate=24% last=E5120 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv --exempt Toys,garden` inside the project folder and respond with just the 8-character code it prints after `checksum`.

    answer

    answer hidden on shared reports

    agent's debrief

    Implemented --exempt by parsing argv in main.py and adding an EXTRA_EXEMPT set consulted in skips_tax; verified the request example prints gross=36.00 exactly as specified, unit tests still pass, and data/sample.csv without the option still prints the README checksum 6714f15c. Straightforward once the module layout was read; no ambiguity except where to thread the option, which I kept simple with a module-level set.

  • spec-violation-1✓ pass2m 27s

    prompt

    Download airbench.ai/f/975114f513a57f28f7b6f959481afd8c.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.

    answer

    answer hidden on shared reports

    agent's debrief

    Audited README rules against code; the off-by-one was tier_pct using qty>30/qty>8 while README says 30-or-more/8-or-more, so qty 8 and 30 got no or the wrong discount. Fixed to >=; the unit tests did not pin the boundaries. Confirmed by data/sample.csv now printing exactly the README checksum 6f4d4eec, then used real.csv output.

  • far-cause-1✓ pass35s

    prompt

    Download airbench.ai/f/d395320dc782a49371889f6ebd12e12d.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `US 2025-11 USD` the ledger has a gross of `42771.65`, but the report prints `gross=40014.74`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.

    answer

    answer hidden on shared reports

    agent's debrief

    The gross mismatch was far from its cause: checkout/clock.py parse_ts returned local.replace(tzinfo=utc) + offset, which moves offset timestamps the wrong way by twice the offset. That corrupted UTC instants, changing revision resolution under rule 4 and which rows landed in US 2025-11. Fix is minus offset. The sample has no offset timestamps so its checksum is unaffected; after the fix sample still prints d5afb148 and real.csv US 2025-11 gross matches the ledger 42771.65.

  • interacting-1✓ pass1m 05s

    prompt

    Download airbench.ai/f/6756649cb65ba5c0df6c706bd180a406.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `LATAM 2025-12 USD` | `73189.88` | `72759.07` | > | `LATAM 2026-01 JPY` | `1325306` | `12645.46` | > | `US 2025-11 USD` | `423603.53` | `410662.94` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.

    answer

    answer hidden on shared reports

    agent's debrief

    Three interacting README violations, none exercised by sample.csv or the unit tests. 1) reader.clean_row used int(qty or 0) so blank quantities priced as zero instead of one. 2) DECIMALS had JPY=2 with a compensating //=100 in to_minor; format_major then printed JPY with 2 decimals (12645.46 vs ledger 1325306). Set JPY=0 and removed the hack. 3) revisions.keep_latest kept the first row per (order_id,sku) instead of the latest instant with later-in-file tie-break, understating US 2025-11 and LATAM groups. After all three fixes sample still prints README checksum 83113bbe and the three flagged groups match the ledger exactly.

  • perf-1✓ pass3m 33s

    prompt

    Download airbench.ai/f/fca632a4632e21cb9467c1da13a28c4b.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 1147326970 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.

    answer

    answer hidden on shared reports

    agent's debrief

    Four quadratic hotspots replaced: dupes.keep_latest scanned a list per event (list.index) -> dict of event_id to kept-list index; membership.join_users linear-scanned the user list per event -> dict user_id to plan (setdefault keeps first U line); rollup.session_size scanned all events per long session -> len(s.sessions events) since session runs are contiguous per user; peaks.peak_overlap tested every session against every session -> sort start(+1)/end(-1) points, starts before ends at equal time, sweep for max live. Sample still prints README checksum 35c858ef; data/big.log (gen.py 1147326970 1000000, 1M events) runs in ~7s.

  • rev-cpu-target1— unanswered—

    prompt

    This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/d2e1fee0ded69c66f45f53cd0712bc21.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>
  • rev-cpu-target2— unanswered—

    prompt

    This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/d2e1fee0ded69c66f45f53cd0712bc21.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>
  • rev-cpu-target3— unanswered—

    prompt

    This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/d2e1fee0ded69c66f45f53cd0712bc21.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>
  • rev-cpu-target4— unanswered—

    prompt

    This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/d2e1fee0ded69c66f45f53cd0712bc21.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>
  • rev-cpu-target5— unanswered—

    prompt

    This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/d2e1fee0ded69c66f45f53cd0712bc21.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>
  • rev-cpu-target6— unanswered—

    prompt

    This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/d2e1fee0ded69c66f45f53cd0712bc21.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>
  • opt-route-bronze— unanswered—

    prompt

    This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/9b38d859816585292f66b69975de1f8d.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
  • opt-route-silver— unanswered—

    prompt

    This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/9b38d859816585292f66b69975de1f8d.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
  • opt-route-gold— unanswered—

    prompt

    This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/9b38d859816585292f66b69975de1f8d.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.

how this agent was configured

Hardware: ASUS Ascent GX10 / DGX Spark GB10, 128 GB unified memory, arm64. Model server: Checkpoint ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3_S @ ed59f92 (125B-A6B MoE) on the Strata engine 0.1.59 / Strata 0.2.13 (github.com/Niko1221/Strata release linux-aarch64 cuda13 packages) on one DGX Spark GB10, served by the m64 capsule qwen38-flash-next-strata-gb10 (dh7/capsules): strata launch --family qwen --model IQ3_S --context 65536 --vision on --parallel 1 --tune skip, MTP drafting, default KV. Sampling is the capsule's (Strata's defaults): the m64 gate accepts only messages/tools/tool_choice/response_format/max_tokens/stream/seed, so the checkup proxy drops other request fields (temperature, top_p, ...). Harness: opencode 1.18.23 on the host: `opencode run --auto --format json <prompt>`; per-run opencode.json pins the model via @ai-sdk/openai-compatible; context 65536, max output 16384 tokens, image input declared only when the server passed a two-image probe. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 63fe603; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.

discussion

Sign in to join the discussion

No messages yet.