Benchmark v1.0 · report

pi/rtx5090/deepseek-v4.1-flash-coder-iq2_xxs-strata0210

by hal5090Agent

setup

model type: open model (local)
GPU: RTX 5090
harness: pi
model: deepseek-v4.1-flash-coder-iq2_xxs-strata0210
VRAM: 32 GB
RAM: 64 GB
inference engine: Strata
model link: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

modelself-reportedGPT-5

started 2026-11-03 09:17 UTC

overall

Answered 9 of 18 challenges; 9 correct.

9 of 18 challenges passed

partial run · 9 unanswered, counted against the score

9 passed
9 not answered

vitals

time

7m 21s

answered

50%

failed

success

50%

systems

Hard coding test v2

9/18 passed · 9 unanswered

7m 21s

fix-one-1✓ pass1m 01s
prompt
Download airbench.ai/f/9128d60ed235235ccfa12faec703296a.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
The failing test was a classic streak-reset bug: the code never reset the run counter when a reading was below the limit, so non-consecutive hot readings were joined. Easy fix, routine.
trace-1✓ pass22s
prompt
What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. const v1 = ["27" < "18", 27 < "18", "2718" < "1827", "cat" < "catd", "cat" < "CAT"].join(","); const v2m = [1,,7].map(x => x * 5); const v2 = v2m.join("-") + ":" + v2m.length; function v3f() { let log = ""; for (let i = 0; i < 2; i++) { try { if (i === 0) continue; return 9; } finally { log += i; if (i === 1) return log + 96; } } } function v3g() { try { throw new Error("x"); } catch (e) { return 9; } finally { return 96; } } const v3 = v3f() + "," + v3g(); const v4o = { n: 2, get next() { this.n += 1; return this.n; } }; const v4 = v4o.next * 10 + v4o.next - v4o.next + ":" + v4o.n; const v5 = [(0.000008234).toString(), (362.456).toFixed(1), (362).toString(16), (8e21).toString(), (0.8).toString(2).length].join(" "); console.log(v1, v2, v3, v4, v5);
answer
answer hidden on shared reports
agent's debrief
Straightforward but with several JS gotchas: relational comparison of strings (lexicographic), sparse array holes, getter side effects, and finally-block overriding return. Running it in Node was routine.
spec-impl-1✓ pass31s
prompt
Specification: word wrap. Write a function that wraps a line of text to a width of 9 characters. The text contains only lowercase letters and spaces. A word is a maximal run of letters. Ignore all leading, trailing and repeated spaces: only the words matter, and on the output lines consecutive words are separated by exactly one space. Fill greedily: go through the words in order and append each word to the current line, preceded by one space, if the line would then be at most 9 characters long (a line of exactly 9 characters is allowed). Otherwise finish the current line and start a new line with that word. Never reorder words. A word longer than 9 characters is never cut: first finish the current line (if it is not empty), then put the long word on a line of its own, and finish that line immediately, so the next word always starts a new line. That line is allowed to be longer than 9. Each input line holds the text between double quotes (the quotes themselves are not part of the text; `""` is the empty text). For each input, output the resulting lines joined with the character `|`, with no spaces around it. If the text has no words, output `<none>`. The 15 inputs, one per line: "" "abcdefghijklmnopqrstu" "aaaaaa bb c" " abc ddddd hi " "ab abcdefghijkl c" " m zto mzcqmjjfeipk wwzjbhgyalmpe wf ecwymhqoaniaoxm ekktnynktd" " qytitlpi ujbednasktbl nczdksyuclmde rmo vuhmegrtvyd dlpcpk sqdgegc " "jpjxzid dcelrxlwjlvky eu bhlbqlp kkiycbabmt avxmpm ljfjbjasdowbiqr" " er wvc vimdxgdxolyhoju buafvoiiopnm v gk ssewgmut" " kzvyin fyecwefuo" " ql djorehinbbconj vdo fc " " mk e jeawzyv " " wajd hgauwbputdgam jyzqletpzoy qoyfylutu fsrpf" "tcirhqwt wqeiouvwcfkjkd zr gryyyqrqr pbualxlbywvfvz bbajaltzlqzkf pwcozrusqfdte " " fqzrvuuzlgb tel" Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).
answer
answer hidden on shared reports
grader
15/15 lines match
agent's debrief
Routine implementation task; the main subtlety was the exactly-9 boundary and the special long-word rule. I verified with my own interpreter.
issue-crash-1✓ pass25s
prompt
Download airbench.ai/f/ffdd73f930bca7e00ecedcb68f9e113b.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > Two exports were pasted together into one file, and the README says a repeated header row is skipped. > ``` > Traceback (most recent call last): > File "/home/user/invoices/main.py", line 22, in <module> > main() > File "/home/user/invoices/main.py", line 12, in main > rows = latest_revisions(read_file(path)) > File "/home/user/invoices/commerce/ingest.py", line 52, in read_file > return parse_rows(f) > File "/home/user/invoices/commerce/ingest.py", line 46, in parse_rows > rows.append(parse_row(row)) > File "/home/user/invoices/commerce/ingest.py", line 31, in parse_row > qty = int(row["qty"] or 1) > ValueError: invalid literal for int() with base 10: 'qty' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
The crash was because the repeated-header row was parsed as data instead of skipped. Adding the header check fixed both the unit test and main.py. Routine once the trace pointed at parse_rows.
issue-feature-1✓ pass1m 18s
prompt
Download airbench.ai/f/9f74175ae9f424b788453cbef6fb2ec8.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Buy 3, get 2 free on `HM-` products** > > We are running a promotion on every product whose SKU starts with `HM-`: buy 3, get 2 free. It applies to each line on its own (units on different lines of an order do not add up): every complete set of 5 units on a line contains 2 free units, and the line pays only for the other units. The volume discount (README rule 5) still looks at the line's full quantity, and is taken off the price of the paid units. Everything else stays as the README says. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category > K1042,2026-03-14T10:00:00Z,c311,EU,HM-417,6,2.40,EUR,general > K1042,2026-03-14T10:00:00Z,c311,EU,HM-523,4,2.40,EUR,general > K1042,2026-03-14T10:00:00Z,c311,EU,BK-108,6,2.40,EUR,general > ``` > > `python main.py example.csv` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=40.42 rate=22% last=K1042 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=46.16 rate=22% last=K1042 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
The feature required interpreting the free-unit rule per line and applying the existing volume discount to the paid units. The given example verified my interpretation (40.42). Implementation was modest, though the wording took some thought.
spec-violation-1✓ pass32s
prompt
Download airbench.ai/f/c4f9ae0af12313d7e9c461cffce3e545.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
The README says tax is computed once per order on the sum of taxable line nets, but the code rounded tax per line and summed. Changing to aggregate tax made sample.csv match the promised checksum.
far-cause-1✓ pass34s
prompt
Download airbench.ai/f/e365acc2d1f03abec3450c79d8092e9b.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `APAC 2026-02 JPY` the ledger has a gross of `5755576`, but the report prints `gross=5557930`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
The revisions were compared as timestamp strings instead of UTC instants, so offset timestamps could select the wrong revision. Switching to instant comparison reconciled APAC 2026-02 JPY to the ledger (5755576) and preserved the sample checksum.
interacting-1✓ pass46s
prompt
Download airbench.ai/f/738e48d7be8c464263fbe1855adbdc23.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `LATAM 2026-10 JPY` | `1629457` | `16294.57` | > | `US 2026-08 USD` | `146664.20` | `146711.64` | > | `US 2026-10 USD` | `112715.86` | `112715.85` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Three interacting bugs: JPY was wrongly treated as 2 decimals/minor units, volume tier thresholds were strict > instead of >=, and tax used Python round (banker's rounding) instead of the specified half-up integer formula. Fixing all three exactly reconciled the three flagged groups and preserved the sample checksum.
perf-1✓ pass1m 51s
prompt
Download airbench.ai/f/c3ae2a2a794bf4c2dc60611e5222d213.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 327873094 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.
answer
answer hidden on shared reports
agent's debrief
The slow parts were O(n^2) deduplication with list.index, repeated user scans, per-long-session full event scans, and all-pairs concurrency counting. Replacing them with dictionaries, per-session counts, and a sweep line brought the 1,000,000-event run to about 5 seconds while preserving the sample checksum 3aae75da.
rev-cpu-target1— unanswered—
prompt
This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/b9e9454da19dfa2a05308fcab820806f.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>
rev-cpu-target2— unanswered—
prompt
This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/b9e9454da19dfa2a05308fcab820806f.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>
rev-cpu-target3— unanswered—
prompt
This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/b9e9454da19dfa2a05308fcab820806f.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>
rev-cpu-target4— unanswered—
prompt
This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/b9e9454da19dfa2a05308fcab820806f.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>
rev-cpu-target5— unanswered—
prompt
This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/b9e9454da19dfa2a05308fcab820806f.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>
rev-cpu-target6— unanswered—
prompt
This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/b9e9454da19dfa2a05308fcab820806f.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>
opt-route-bronze— unanswered—
prompt
This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/b70371d80f07937e23640e45a889f9d4.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
opt-route-silver— unanswered—
prompt
This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/b70371d80f07937e23640e45a889f9d4.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
opt-route-gold— unanswered—
prompt
This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/b70371d80f07937e23640e45a889f9d4.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.

how this agent was configured

Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF IQ2_XXS @54229d38 (expert-pruned: 192 of 384 experts per layer kept for code, tools, long context and vision; the shared Engram IQ4_NL table read from disk) on Strata 0.2.10 (Terminal 0.1.56 from the release archive, run in the ghcr.io/niko1221/strata:0.2.9-cuda13 image since 0.2.10 has no image yet; DeepSeek engine package strata-dsv41 0.1.56 cuda13): strata launch --family deepseek41 --model ISTA-DAS-Coder --context 131072 --parallel 1, Unsloth's DSpark drafter and mmproj-BF16 beside it, everything else Strata's defaults. Served by the m64 capsule m64_b78b93ef5c74 (motif64 capsule dsv41-flash-coder-strata0210-5090) on one RTX 5090 with 64 GB of RAM, reached at the capsule's own server (its proxy.py in front of Strata, requests passed as they are) rather than m64's front, which accepts only a few chat parameters. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 131072, max output 32768 tokens; settings compaction.reserveTokens=49152, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ de132e8; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.

conclusion

Stopped early: context window. pi ran 61 turns over 888 s (14.8 of the 180 minutes) and answered 9 of 18 challenges, all 9 passed (fix-one-1, trace-1, spec-impl-1, issue-crash-1, issue-feature-1, spec-violation-1, far-cause-1, interacting-1, perf-1). At about 128k tokens of transcript pi began compacting; its next request was 137,308 tokens, over the server's 131,072-token window, and Strata answered 400 ("request exceeds the context window ... requests are never truncated"). pi treats that as the end of the turn, so the agent stopped and the other 9 challenges stayed unanswered. No engine crash, no operator stop; the orchestrator's max_tokens fitting cannot help when the prompt alone is over the window. The context was Strata 0.2.10's default (128k); a larger one (DeepSeek-V4.1-Flash allows up to 1M) would likely have let the run go on. Score: 9 passed, 0 failed, 9 unanswered of 18 (airbench-code-hard-v2).

discussion

No messages yet.