Benchmark v1.0 · report

pi/rtx5090/deepseek-v4.1-flash-siege-ladder

setup

model type: open model (local)
GPU: RTX 5090
harness: pi
model: deepseek-v4.1-flash-siege-ladder
VRAM: 32 GB
RAM: 64 GB
inference engine: Siege-Ladder
model link: https://huggingface.co/MasherDon/DeepSeek-V4.1-Flash-1017-SiegeLadder-GGUF

modelself-reportedpi-agent

started 2026-11-08 00:48 UTC

this run failed · 2026-11-08 03:17 UTC

Answered 9 of 18 challenges, and all 9 passed: fix-one, trace, spec-impl, issue-crash, issue-feature, spec-violation, far-cause, interacting, perf. The run ended after 148 of the card's 180 min, when a request of 133,800 tokens overflowed the 131,072-token window (a 400 from llama-server). pi then started compacting the context and exited before the compaction finished. Siege-Ladder rereads a new prompt at only about 166 tok/s, so compacting at this size takes about 10 min. Decode ran at about 17-25 tok/s with experts streamed from NVMe. Everything it submitted was correct; it was speed and the context window, not accuracy, that cut the score.

overall

Answered 9 of 18 challenges; 9 correct.

9 of 18 challenges passed

partial run · 9 unanswered, counted against the score

9 passed
9 not answered

vitals

time

failed

answered

50%

failed

success

50%

systems

Hard coding test v2

9/18 passed · 9 unanswered

failed

fix-one-1✓ pass2m 52s
prompt
Download airbench.ai/f/f0ec18158a82aabd8c6d100b9cab367e.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Straightforward unit-test bug: the bad-quantity counter was never incremented. The only failing test pointed right at it.
trace-1✓ pass1m 52s
prompt
What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. const v1m = [2,,8].map(x => x * 2); const v1 = v1m.join("-") + ":" + v1m.length; function v2f(a, b) { arguments[0] = 14; b = 15; return [a, arguments[1], arguments.length].join("/"); } const v2 = v2f(6, 7) + ";" + v2f(6); function v3f() { let log = ""; for (let i = 0; i < 2; i++) { try { if (i === 0) continue; return 1; } finally { log += i; if (i === 1) return log + 53; } } } function v3g() { try { throw new Error("x"); } catch (e) { return 1; } finally { return 53; } } const v3 = v3f() + "," + v3g(); let v4 = ""; switch (3) { case 5: v4 += "a"; case 3: v4 += "b"; case 4: v4 += "c"; break; default: v4 += "d"; case 2: v4 += "e"; } function v5make(n) { return () => (n += 3); } const v5a = v5make(4); const v5b = v5make(20); const v5 = [v5a(), v5b(), v5a(), v5a(), v5b()].join(","); console.log(v1, v2, v3, v4, v5);
answer
answer hidden on shared reports
agent's debrief
Routine JS semantics puzzle; I verified with node. Sparse-array map, mapped arguments, finally overriding return, switch fallthrough and closures were all as expected.
spec-impl-1✓ pass1m 14s
prompt
Specification: tiered pricing. Write a function price(units) that returns the cost, in whole cents, of buying `units` units (a non-negative integer) under marginal pricing: - the first 7 units cost 85 cents each; - units 8 through 33 (inclusive) cost 45 cents each; - every unit after the 33th costs 40 cents each. Each bracket's rate applies only to the units that fall inside that bracket (it is not retroactive to earlier units). Add the three bracket subtotals, then round the total to the nearest multiple of 10 cents. A total that ends in exactly 5 sits halfway between two multiples of 10: round it down (for example 1235 becomes 1230). Any other total rounds to the closer multiple of 10. Zero units cost 0. For each input line, which holds `units`, output the rounded total as a plain integer number of cents (no currency symbol, no decimal point). The 15 inputs, one per line: 0 7 8 34 33 62 45 11 56 73 56 31 21 48 66 Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).
answer
answer hidden on shared reports
grader
15/15 lines match
agent's debrief
Easy arithmetic spec. The only subtlety was the ties-round-down rule, and conveniently every raw total here ended in 5, so all got rounded down by 5. I double-checked bracket boundaries (7/8 and 33/34).
issue-crash-1✓ pass2m 30s
prompt
Download airbench.ai/f/74207ee97abbaacc9338192883cbd757.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > A few rows have currencies like "eur" or " USD ", and the README says spaces are ignored and case does not matter for currencies. > ``` > Traceback (most recent call last): > File "/home/user/invoices/main.py", line 22, in <module> > main() > File "/home/user/invoices/main.py", line 12, in main > rows = drop_duplicates(read_file(path)) > File "/home/user/invoices/checkout/ingest.py", line 28, in read_file > return read_rows(f) > File "/home/user/invoices/checkout/ingest.py", line 37, in read_rows > rows.append(clean_row(row)) > File "/home/user/invoices/checkout/ingest.py", line 46, in clean_row > raise ValueError(f"unknown currency {cur!r}") > ValueError: unknown currency ' JPY ' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Simple one-line fix: normalize currency with strip().upper() before lookup. Sample checksum then matched the README, which gave me confidence nothing else was broken.
issue-feature-1✓ pass3m 13s
prompt
Download airbench.ai/f/636f52938e97ffa5467f6df5a41e5192.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Make more categories tax-exempt from the command line** > > Please add an option `--exempt` that takes a comma-separated list of categories that are tax-exempt in addition to books, for example `--exempt Toys,general`. Category names on the command line are case-insensitive, like categories in the file. Books stay exempt. Without the option nothing changes. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category > E5120,2026-03-14T10:00:00Z,c650,EU,TY-233,1,6.00,EUR,toys > E5120,2026-03-14T10:00:00Z,c650,EU,HM-402,2,6.00,EUR,general > E5120,2026-03-14T10:00:00Z,c650,EU,BK-115,3,6.00,EUR,books > ``` > > `python main.py example.csv --exempt Toys,general` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=36.00 rate=18% last=E5120 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=39.24 rate=18% last=E5120 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv --exempt Toys,general` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Feature was easy to plumb: added an extra exempt set through order_tax/price_order/build_orders and a small argv parser. Verified the sample checksum was unchanged and the issue example printed exactly the expected lines.
spec-violation-1✓ pass2m 29s
prompt
Download airbench.ai/f/05344f766d3e01670620b69564077da7.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Found it by reading README rule 7 against levies.py: tax was being rounded per line instead of once on the summed taxable net. After fixing, the sample checksum matched the README exactly.
far-cause-1✓ pass2m 07s
prompt
Download airbench.ai/f/6c7a6e5017ad61ffb0bad6e33e8bf416.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `LATAM 2025-11 JPY` the ledger has a gross of `1739720`, but the report prints `gross=173971106`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
The far cause was a shadowed DECIMALS table in reader.py that set JPY to 2 decimals, so yen prices were multiplied by 100. Removing the duplicate and importing the real table fixed it; the LATAM group then matched the ledger and the sample checksum matched the README.
interacting-1✓ pass2m 48s
prompt
Download airbench.ai/f/0f1c405db43a91dc8d05cc7b15344796.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `APAC 2026-04 JPY` | `2275308` | `22877.33` | > | `EU 2026-06 EUR` | `423259.64` | `423586.42` | > | `US 2026-06 USD` | `220273.31` | `220478.28` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.
answer
answer hidden on shared reports
agent's debrief
Three interacting causes: JPY was declared with 2 decimals plus a /100 hack, discount thresholds used > instead of >=, and category exemption did not lowercase. After fixing all three, every ledger figure in the issue matched and the sample checksum matched the README.
perf-1✓ pass5m 53s
prompt
Download airbench.ai/f/d681ab8af91ca6179cbe73da7fe57a36.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 550681378 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.
answer
answer hidden on shared reports
agent's debrief
Replaced the quadratic hot spots (dedupe via dict, user join via plan dict, pair counting via set, concurrency via a start/end sweep) while keeping semantics. Verified byte-identical output on a medium generated log and the sample checksum f11ad258; big.log then ran in a few seconds.
rev-cpu-target1— unanswered—
prompt
This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/180072ace8c866f051c7cb4aa6cd7859.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>
rev-cpu-target2— unanswered—
prompt
This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/180072ace8c866f051c7cb4aa6cd7859.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>
rev-cpu-target3— unanswered—
prompt
This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/180072ace8c866f051c7cb4aa6cd7859.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>
rev-cpu-target4— unanswered—
prompt
This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/180072ace8c866f051c7cb4aa6cd7859.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>
rev-cpu-target5— unanswered—
prompt
This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/180072ace8c866f051c7cb4aa6cd7859.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>
rev-cpu-target6— unanswered—
prompt
This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/180072ace8c866f051c7cb4aa6cd7859.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>
opt-route-bronze— unanswered—
prompt
This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/23df42b731e932596e9cd28e9a7e9042.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
opt-route-silver— unanswered—
prompt
This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/23df42b731e932596e9cd28e9a7e9042.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
opt-route-gold— unanswered—
prompt
This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/23df42b731e932596e9cd28e9a7e9042.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.

how this agent was configured

Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint MasherDon/DeepSeek-V4.1-Flash-1017-SiegeLadder-GGUF og build (378 GB: own GPTQ of the 40 layers, IQ2_XXS gate/up and IQ3_XXS down experts, Q5_K dense, Engram tables, the G44 lens) on Siege-Ladder (github.com/MasherDon/Siege-Ladder @ 06ffbac: 121 patches over JigSawPT's llama.cpp dsv41-porte @ 8ce0968, built with CUDA 13.0 for sm_120, plus homelab-inference machines/hal/siege-llama-dsml-tool-markup.patch so the model's space-prefixed DSML tool markup is parsed as tool calls): llama-server -ngl 99 -t 12 --parallel 1 --jinja --moe-stream-direct --moe-stream-l2 36 (pinned RAM tier, GiB) -c 131072 -b 32768 -ub 8192 -ctk q8_0 -ctv q8_0, expert VRAM cache auto, default preset; reasoning from the chat template. Experts streamed VRAM -> pinned RAM -> NVMe. Research code measured on Windows by its author; first Linux run here. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 131072, max output 32768 tokens; settings compaction.reserveTokens=49152, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ f51d1fc; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.

conclusion

discussion

No messages yet.