Benchmark v1.0 · report
pi/rtx5090/qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- pi
- model
- qwen3.8-flash-next-ud-iq4_xs-unsloth-v3.1
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- llama.cpp
started 2026-11-04 23:05 UTC
overall
Answered 9 of 18 challenges; 9 correct.
9 of 18 challenges passed
partial run · 9 unanswered, counted against the score
- 9 passed
- 9 not answered
vitals
time
10m 06s
answered
50%
failed
0%
success
50%
systems
Hard coding test v2
9/18 passed · 9 unanswered
fix-one-1✓ pass1m 22s
prompt
Download airbench.ai/f/18d7d729449da0c80f902e928942e370.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Straightforward: the failing test pinned the bug exactly (grade_letter used > instead of >= at the 800 boundary, so 800 fell through to C). One-line fix, tests green, checksum read off program output. Only mild ambiguity was whether 'inside the folder' meant the zip root or the grades/ subpackage, but output is identical either way.
trace-1✓ pass32s
prompt
What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. function v1f(x = y, y = 58) { return x + y; } const v1safe = (fn) => { try { return fn(); } catch (e) { return "TDZ"; } }; const v1 = [v1f(3), v1safe(() => v1f()), v1f(3, 4), v1safe(() => v1f(undefined, 3))].join(","); function v2f() { let log = ""; for (let i = 0; i < 2; i++) { try { if (i === 0) continue; return 9; } finally { log += i; if (i === 1) return log + 27; } } } function v2g() { try { throw new Error("x"); } catch (e) { return 9; } finally { return 27; } } const v2 = v2f() + "," + v2g(); function* v3gen() { for (let i = 1; i <= 5; i++) yield i * 4; } const v3it = v3gen(); const v3head = []; for (let i = 0; i < 2; i++) v3head.push(v3it.next().value); const v3ret = v3it.return(93); const v3 = v3head.join("+") + "|" + v3ret.value + "|" + v3ret.done + "|" + [...v3it].length; const v4m = [8,,1].map(x => x * 3); const v4 = v4m.join("-") + ":" + v4m.length; function v5make(n) { return () => (n += 2); } const v5a = v5make(4); const v5b = v5make(17); const v5 = [v5a(), v5b(), v5a(), v5a(), v5b()].join(","); console.log(v1, v2, v3, v4, v5);answer
answer hidden on shared reportsagent's debrief
I traced all five groups by hand first (parameter-scope TDZ on x=y, finally-block return overriding both continue and return, generator .return() leaving the iterator done, map() preserving holes so join emits an empty field, and two independent n+=2 closures) and my prediction matched node's output character for character. The trickiest bit was v2f: reasoning that the finally-return at i=1 wins over the try's return 9 while the i=0 finally still ran before the continue, giving '0127'. Having node locally made this a confidence check rather than a gamble.
spec-impl-1✓ pass32s
prompt
Specification: version comparison. Write a function that compares two version strings a and b. Each version is one or more non-negative integers (no leading zeros) joined by `.`, optionally followed by one pre-release suffix: `-beta` and then an integer N, written directly after the last number (for example `2.0-beta7`). The beta in the suffix is always lowercase; N has no leading zeros. Rules, applied in this order: 1. Compare the numeric parts from left to right as integers, not as text (1.10 is greater than 1.9). The first difference decides, and the suffix is ignored in this step. 2. If all the parts the two versions have in common are equal but one has more parts: The version with fewer numeric parts is the lesser one, even if the extra parts of the other are 0 (so 1 is less than 1.0). If this rule decides the comparison, the suffixes are ignored. 3. If the numeric parts are equal after steps 1 and 2, a version with a pre-release suffix is less than the same version without one. If both have a suffix, compare their N values as integers. If neither has a suffix, or both have the same N, the versions are equal. For each input line, which holds two versions `a b` separated by one space, output `<` if a is less than b, `=` if they are equal, or `>` if a is greater than b. The 15 inputs, one per line: 1.0 1 1.10 1.9 2.0-beta1 2.0 2.0-beta10 2.0-beta9 10.8.7-beta4 10.8.7 5.0.9-beta4 5.0.9.0-beta4 12-beta8 12.0-beta8 11.1.7 11.1.7.0.0 1.4-beta7 1.4-beta7 8.7.4 8.7.4-beta2 2.9.1 2.9.1.0.0 3.2.0 3.2 7 7-beta5 7 7.0.0 1.0.9-beta10 1.0.9 Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).answer
answer hidden on shared reportsgrader
15/15 lines match
agent's debrief
Implemented the three rules in the stated order with a small regex parser plus integer part lists, and hand-checked every line against the spec before running it. The cases that actually bite are rule 2's 'fewer parts always wins even with zeros' (1.0 vs 1, 7 vs 7.0.0, 12-beta8 vs 12.0-beta8) because it deliberately contradicts the usual semver behaviour, and the interaction where rule 2 decides before suffixes are ever consulted (5.0.9-beta4 vs 5.0.9.0-beta4). Nothing blocked me; I am reasonably confident, though rule 2's wording 'equal after steps 1 and 2' left a moment of doubt about whether a suffix could rescue the shorter version - I took the explicit 'the suffixes are ignored' sentence at face value.
issue-crash-1✓ pass37s
prompt
Download airbench.ai/f/706cd99855703bd85da6c70af4f83832.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > Some products are in categories the report has never heard of, and the README says every category other than books is taxed. > ``` > Traceback (most recent call last): > File "/home/user/shipments/main.py", line 22, in <module> > main() > File "/home/user/shipments/main.py", line 13, in main > orders = group_orders(rows) > File "/home/user/shipments/retail/order_book.py", line 42, in group_orders > settle_order(order) > File "/home/user/shipments/retail/order_book.py", line 26, in settle_order > order.tax = order_tax(order.lines, order.region) > File "/home/user/shipments/retail/vat.py", line 21, in order_tax > if not is_exempt(line.category): > File "/home/user/shipments/retail/vat.py", line 12, in is_exempt > kind = CATEGORY_KIND[cat] > KeyError: 'garden' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Crash was exactly where the traceback said: vat.is_exempt did a dict lookup CATEGORY_KIND[cat] over a whitelist of three categories, while the README says the exemption rule is binary - only 'books' is exempt, everything else is taxed. One-word fix to .get(cat, 'taxed'), which also keeps the sample checksum 5c9fff4e intact, so it was clearly the intended change. Routine and satisfying; the only thing I'd flag is that the whitelist style invites the same bug again for regions/rates, though those already use .get correctly.
issue-feature-1✓ pass55s
prompt
Download airbench.ai/f/bec13e2a07a96efd3afe2c3b96f5ae2c.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Small-order fee** > > Orders whose net (the sum of their line nets, README rule 5) is below 3000 minor units get a small-order fee of 350 minor units (minor units are cents for EUR and USD, so 30.00 and 3.50, and yen for JPY). The fee becomes part of the order's net and is taxed at the order's rate, even when every line of the order is exempt; the gross is still net plus tax. The fee is added to the order's taxable amount from README rule 7, so the tax is still rounded once per order. Orders of 3000 minor units or more pay no fee. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category > S7310,2026-03-14T10:00:00Z,c118,US,HM-640,1,28.99,USD,general > S7311,2026-03-14T10:00:00Z,c529,EU,HM-640,1,30.00,EUR,general > ``` > > `python main.py example.csv` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=36.90 rate=23% last=S7311 > US 2026-03 USD orders=1 gross=36.06 rate=11% last=S7310 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-03 > EU 2026-03 EUR orders=1 gross=36.90 rate=23% last=S7311 > US 2026-03 USD orders=1 gross=32.18 rate=11% last=S7310 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Feature request was precise and the worked example was the real spec: I checked my reading against it before writing code (US 28.99 -> net 2899 + 350 fee = 3249, tax (3249*11+50)//100 = 357, gross 36.06) and my first implementation reproduced both expected report lines exactly, which is strong evidence the fee belongs inside net and inside taxable. Kept order_tax's fee argument defaulted so the existing tax tests still pass; the existing 15 tests stayed green and the sample checksum moved off 6af5103c as the prompt said it should. Slight uncertainty remains on whether 'below 3000' is strictly less-than (I used <, so the 3000 EUR order pays nothing) - the example's EU line confirms that boundary.
spec-violation-1✓ pass1m 09s
prompt
Download airbench.ai/f/94adb8ecc9dc09f91ed27ba29ccff3d8.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Hardest of the three 'fix the project' tasks so far because nothing points at the bug - all 15 tests were green by design, so I had to read every module against the README. The giveaway was in the data: sample.csv contains 'BOOKS' and 'Books' categories, and tax.skips_tax did CATEGORY_KIND.get(category) without lowering, so capitalised books got taxed while the README says category case does not matter. One-line fix and the sample checksum jumped from 59f3fa7c to the README's 46954682, which is the strongest confirmation this kind of task offers. I also walked real.csv's category/region/currency distributions to be sure it holds no further surprises the sample never exercises (lowercase currencies and unknown categories are already handled correctly), so I am fairly but not totally confident real.csv has no second, unexercised violation.
far-cause-1✓ pass44s
prompt
Download airbench.ai/f/db09d0c707c11959d64f80ae307e8efa.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `US 2026-03 USD` the ledger has a gross of `177071.39`, but the report prints `gross=177070.04`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
The 'far cause' framing was accurate - the bug is nowhere near the reported symptom. I read every module against the README and the one deviation was ingest.to_record defaulting a blank qty to 0 instead of 1. It hid in the sample because sample.csv happens to contain no blank-quantity rows at all (26 in real.csv), which is exactly why the README checksum still passed. Best part of this challenge was the built-in check: after the fix the US 2026-03 USD group came out at 177071.39, matching finance's ledger to the cent, so I am confident rather than merely hopeful.
interacting-1✓ pass1m 31s
prompt
Download airbench.ai/f/fbbbb26cbe9c38139db9de01bfb66b42.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `APAC 2026-03 JPY` | `3743814` | `36243.14` | > | `EU 2026-04 EUR` | `708864.84` | `708577.89` | > | `LATAM 2026-05 USD` | `99675.43` | `99609.19` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Hardest of the README-fix tasks: the issue said 'there may be more than one cause' and there were three, and they had to interact to produce those three numbers. (1) money.DECIMALS had JPY:2 with a compensating '//=100' hack in to_minor - so parsing was right but formatting printed 36243.14 for a yen amount, which is what tipped me off; (2) tax used round(taxable*rate/100) instead of the README's (taxable*rate+50)//100, i.e. banker's rounding on a float instead of integer half-up; (3) revisions.keep_latest ordered revisions by comparing the raw ts STRINGS, so a '+09:00' revision could beat a later UTC instant. Fixing all three reproduces finance's ledger exactly on all three quoted groups (3743814 / 708864.84 / 99675.43) and leaves the sample checksum 6bd686b9 untouched, which is consistent with the sample exercising none of them. I verified with a script that the ts-string dedupe really diverges on 30 rows and blank-qty rows exist, so I am not carrying a fourth undiscovered bug - though I cannot rule one out for groups finance never quoted.
perf-1✓ pass2m 43s
prompt
Download airbench.ai/f/43d8f3446aede0744c6bb11843db9ee4.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 303905495 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.answer
answer hidden on shared reportsagent's debrief
Fun one - four separate quadratic/hot-path sins: dedupe_events did list.index() on a growing list of event_ids (O(n^2)); attach_plans linear-scanned the whole user list for every event (1e6 x 2e5); leaderboard recomputed max() over every user after every single event; and peak_live asked 'how many sessions are live at t' by scanning all sessions for each session start. I fixed each with the standard equivalent (dict index, plan dict keeping the first U line, incremental argmax that only ever moves to the just-updated user, and a closed-interval sweep line where starts are added before ends that merely tie). Verification was the part I cared about: sample still prints 463ca0ff, and I kept the original package side by side and diffed full reports against it on two 25k/40k-event logs - byte identical, 25.0s -> 0.07s. Big log now runs in ~4.7s. Slight residual risk that some pathological case the sample and my two random logs missed would diverge, but each rewrite is a provably-equivalent algorithm rather than a heuristic.
rev-cpu-target1— unanswered—
prompt
This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/edba8e575c5edc217636e8a0b67a0122.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>rev-cpu-target2— unanswered—
prompt
This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/edba8e575c5edc217636e8a0b67a0122.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>rev-cpu-target3— unanswered—
prompt
This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/edba8e575c5edc217636e8a0b67a0122.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>rev-cpu-target4— unanswered—
prompt
This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/edba8e575c5edc217636e8a0b67a0122.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>rev-cpu-target5— unanswered—
prompt
This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/edba8e575c5edc217636e8a0b67a0122.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>rev-cpu-target6— unanswered—
prompt
This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/edba8e575c5edc217636e8a0b67a0122.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>opt-route-bronze— unanswered—
prompt
This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/202814861358cbe57642c4005eadb5b4.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.opt-route-silver— unanswered—
prompt
This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/202814861358cbe57642c4005eadb5b4.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.opt-route-gold— unanswered—
prompt
This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/202814861358cbe57642c4005eadb5b4.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS @d9620119 (Unsloth Dynamic v3.1, 2026-11-04; 4 shards, the 4th the IQ4_NL n-gram table) on Unsloth's llama.cpp prebuild b11985-mix-c116c32 (linux-x64-cuda13-portable) in nvidia/cuda:13.3.1-runtime-ubuntu24.04 + libgomp1: llama-server -c 262144 -np 1 -fa on -ctk q8_0 -ctv q8_0 --jinja --mmproj mmproj-BF16.gguf -md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2, the rest llama.cpp's defaults (--fit places what fits in VRAM, the other experts on the CPU, the GGUF memory-mapped). Served by the m64 capsule m64_516ff942b134 (motif64 capsule qwen38-flash-next-unsloth-v31-5090) on one RTX 5090 with 64 GB of RAM, reached at the capsule's own server rather than m64's front, which accepts only a few chat parameters. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 262144, max output 32768 tokens; settings compaction.reserveTokens=65536, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ de132e8; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
9 passed, 0 failed, 9 unanswered, after 102 minutes of the 180. Every answered challenge was right (fix-one, trace, spec-impl, issue-crash, issue-feature, spec-violation, far-cause, interacting, perf). The run stopped because the context filled up: the last request reached exactly 262144 tokens, llama.cpp shortened max_tokens to the 353 tokens left, the reply ended with finish_reason length holding only thinking, and pi ended the session instead of compacting first. So the 9 unanswered are the window, not wrong answers. For comparison, Strata 0.2.10 with the IQ3_S at 384k passed 11 with pi (11/5/2). Generation was about 40 tok/s on llama.cpp (CPU experts and SSD reads), about 5 times slower than Strata.
discussion
Sign in to join the discussion
No messages yet.