Benchmark v1.0 · report
pi/openrouter/deepseek-v4.1-flash
setup
- model type
- open model (cloud)
- inference host
- OpenRouter
- harness
- pi
- model
- deepseek-v4.1-flash
started 2026-11-02 18:43 UTC
overall
Answered 18 of 18 challenges; 12 correct.
12 of 18 challenges passed
- 12 passed
- 6 failed
vitals
time
31m 57s
answered
100%
failed
33%
success
67%
systems
Hard coding test v2
12/18 passed
fix-one-1✓ pass29s
prompt
Download airbench.ai/f/6fce8fe7a77203455ddcb8979de51817.zip and unzip it. It contains one Python file with a unit test file (Python 3.10 or newer, standard library only). One test fails because of a bug. Fix the bug, then run the Python file that is not the test file (`python <name>.py`) inside the folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Routine: the max_run_above function never reset its run counter after a value at or below the limit. Added an else clause. Tests pass and checksum printed.
trace-1✓ pass7s
prompt
What exactly does this JavaScript program print? Run it in your head or in any JavaScript engine (non-strict mode). Respond with just the printed output. function v1f(x = y, y = 42) { return x + y; } const v1safe = (fn) => { try { return fn(); } catch (e) { return "TDZ"; } }; const v1 = [v1f(1), v1safe(() => v1f()), v1f(1, 2), v1safe(() => v1f(undefined, 1))].join(","); const v2 = "héllo".length + "/" + [..."héllo"].length + "/" + "héllo".split("").length; let v3 = ""; switch (4) { case 3: v3 += "a"; case 1: v3 += "b"; case 4: v3 += "c"; break; default: v3 += "d"; case 2: v3 += "e"; } const v4a = []; const v4b = []; for (let i = 0; i < 3; i++) v4a.push(() => i * 2); for (var v4j = 0; v4j < 3; v4j++) v4b.push(() => v4j * 2); const v4sum = (fs) => fs.reduce((s, f) => s + f(), 0); const v4 = v4sum(v4a) + "/" + v4sum(v4b); let v5 = ""; outer: for (let i = 0; i < 3; i++) { for (let j = 0; j < 5; j++) { if (j === 2) continue outer; v5 += i * 10 + j + " "; } v5 += "|"; } console.log(v1, v2, v3, v4, v5);answer
answer hidden on shared reportsagent's debrief
Ran it in node to be safe. The subtlety was v1f(undefined,1): even though y is supplied, parameter initialization is left-to-right, so x's default referencing y hits the TDZ. Output ends with a trailing space from v5.
spec-impl-1✓ pass12s
prompt
Specification: interval scheduling. Write a function that, given a list of intervals `s-e` (integers with s < e), selects as many intervals as possible such that no two selected intervals overlap, using this greedy algorithm: 1. Sort all the intervals by end e, smallest first. 2. Intervals with the same end are ordered by earlier start first (smaller s first). Intervals that are still tied (identical intervals) keep their input order. 3. Go through the sorted list once. Select an interval if it does not overlap the interval selected most recently (the first interval is always selected); otherwise skip it. Skipped intervals are never reconsidered. Two intervals overlap only if they share more than a single point: `1-3` and `3-5` do not overlap, because the second starts exactly where the first ends. Each input line holds the intervals separated by single spaces, for example `1-3 4-9`. The word `none` stands for an empty list. For each input, output the number of selected intervals, a colon, then the selected intervals in the order they were selected, separated by commas and with no spaces, for example `2:1-3,4-9`. With no selected intervals output `0:`. The 15 inputs, one per line: none 2-6 2-6 2-6 1-3 3-5 5-8 8-10 1-10 2-3 4-9 6-9 13-17 4-5 20-24 11-17 5-9 0-4 1-7 15-17 5-7 5-11 14-20 11-13 8-13 11-16 7-10 5-9 16-17 7-13 4-5 5-8 9-13 7-10 5-6 13-16 4-5 8-10 11-17 18-23 5-7 11-16 8-9 2-3 20-22 20-25 3-6 0-2 15-18 2-6 20-21 3-4 11-13 9-12 17-23 5-8 3-7 9-14 5-9 1-2 8-14 10-11 4-5 10-14 2-3 2-3 11-15 10-14 17-19 11-15 9-11 20-23 0-4 3-6 20-24 18-24 20-24 5-7 Implement it in any language you like and respond with exactly 15 lines: the result for each input, in order. One result per line, nothing else (no numbering, no commentary).answer
answer hidden on shared reportsgrader
15/15 lines match
agent's debrief
Straightforward greedy implementation. I carefully applied the tie-break (end, then start, then input order) and the half-open overlap rule (start == last end allowed). Hand-verified tricky duplicate/tie cases.
issue-crash-1✓ pass7s
prompt
Download airbench.ai/f/5af2641b426c9e2f19eaae2891958138.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **`python main.py data/real.csv` crashes** > Some products are in categories the report has never heard of, and the README says every category other than books is taxed. > ``` > Traceback (most recent call last): > File "/home/user/receipts/main.py", line 22, in <module> > main() > File "/home/user/receipts/main.py", line 13, in main > orders = assemble_orders(rows) > File "/home/user/receipts/billing/baskets.py", line 40, in assemble_orders > price_order(order) > File "/home/user/receipts/billing/baskets.py", line 26, in price_order > order.tax = order_tax(order.lines, order.region) > File "/home/user/receipts/billing/tax.py", line 21, in order_tax > if not is_exempt(line.category): > File "/home/user/receipts/billing/tax.py", line 8, in is_exempt > kind = CATEGORY_KIND[cat] > KeyError: 'garden' > ``` Fix the bug so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
The bug was is_exempt looking categories up in a fixed dict, so any unknown category raised KeyError. README says only books is exempt and everything else taxed, so I made it a simple equality check. Sample checksum now matches README (45daec31).
issue-feature-1✓ pass28s
prompt
Download airbench.ai/f/c81e27e1bbc10056bbb643b22125aaaf.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this feature request: > **Report per customer (`--group-by customer`)** > > We need the same report per customer. Please add an option `--group-by region|customer`; the default, `region`, is today's report. With `--group-by customer`, orders are grouped by (customer, month, currency) instead of (region, month, currency), the customer takes the region's place as the first field of each line, and the lines are sorted by customer, month, currency. `rate=` then shows the rate of the region of the group's first order (in file order, after revisions). The `period:` line and everything else stay the same. > > For example, with this `example.csv`: > > ```csv > order_id,ts,customer,region,sku,qty,unit_price,currency,category > M3101,2026-03-14T10:00:00Z,c417,US,GN-220,2,12.50,USD,general > M3102,2026-03-14T10:00:00Z,c417,LATAM,HM-905,1,40.00,USD,food > M3103,2026-03-14T10:00:00Z,c208,US,GN-220,1,12.50,USD,general > M3104,2026-04-02T09:30:00Z,c417,US,EL-316,3,8.20,USD,general > ``` > > `python main.py example.csv --group-by customer` should print these report lines (before the checksum): > > ``` > period: 2026-03..2026-04 > c208 2026-03 USD orders=1 gross=13.13 rate=5% last=M3103 > c417 2026-03 USD orders=2 gross=72.25 rate=5% last=M3102 > c417 2026-04 USD orders=1 gross=25.83 rate=5% last=M3104 > ``` > > Today it prints: > > ``` > period: 2026-03..2026-04 > LATAM 2026-03 USD orders=1 gross=46.00 rate=15% last=M3102 > US 2026-03 USD orders=2 gross=39.38 rate=5% last=M3103 > US 2026-04 USD orders=1 gross=25.83 rate=5% last=M3104 > ``` The checksum the README gives for `data/sample.csv` describes the program as it is today; once the request is implemented, the program's output (and that checksum, when run with the request's options) is expected to change. Implement the request, keeping everything else as the README says. Then run `python main.py data/real.csv --group-by customer` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Added --group-by region|customer. Grouping key's first field becomes customer when requested, sorted the same way, and the rate is taken from the region of the group's first order. Verified the feature-request example exactly and confirmed the default sample checksum is unchanged (081f2c9b).
spec-violation-1✓ pass8s
prompt
Download airbench.ai/f/8e2ccc85e8ae73e0e96c33fe39374143.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). The unit tests pass, but `python main.py data/sample.csv` does not print the checksum the README promises. Make the program follow its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
The README says non-listed regions are taxed at 15%, but vat.py had DEFAULT_RATE=0 so LATAM groups were untaxed. Set it to 15; the sample checksum now matches the README (c69887e3).
far-cause-1✓ pass9s
prompt
Download airbench.ai/f/80b7cd5ef448dd299b3ece5f7aff9f90.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report doesn't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. For `APAC 2025-05 JPY` the ledger has a gross of `2928276`, but the report prints `gross=292826885`. > `python main.py data/sample.csv` still prints the checksum in the README. Find the cause and fix it so the program follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Root cause was loader.py defining its own DECIMALS with JPY=2, shadowing the correct currencies.DECIMALS (JPY=0). JPY prices were therefore stored 100x too large and then printed with the 0-decimal formatter, inflating the reported gross. Fixed loader to treat JPY as 0 decimals; APAC 2025-05 JPY now prints 2928276 as the ledger expects.
interacting-1✓ pass18s
prompt
Download airbench.ai/f/9cdf17b421863c1d9afd42b034e39ad4.zip and unzip it. It is a small Python project (Python 3.10 or newer, standard library only). A user filed this issue: > **Report totals don't match the ledger** > Finance reconciled the report for `data/real.csv` against the ledger. These groups are off: > > | group | ledger gross | report prints | > |---|---|---| > | `LATAM 2025-11 JPY` | `2492865` | `22281.26` | > | `LATAM 2025-12 JPY` | `842505` | `7356.74` | > | `LATAM 2025-12 USD` | `243603.34` | `237336.20` | > > All unit tests pass and `python main.py data/sample.csv` prints the checksum in the README, so nothing else flags it. There may be more than one cause. Fix the program so it follows its README, then run `python main.py data/real.csv` inside the project folder and respond with just the 8-character code it prints after `checksum`.answer
answer hidden on shared reportsagent's debrief
Found three interacting bugs: (1) JPY was configured with 2 decimals plus a compensating //100 hack, corrupting formatting; (2) levies DEFAULT_RATE was 0 instead of 17% for unlisted regions; (3) revisions compared timestamp strings instead of UTC instants. After fixing all three, all three ledger groups match and the sample checksum is still 500363c0.
perf-1✓ pass1m 18s
prompt
Download airbench.ai/f/7f60e46e63b87d6852016b18c4d2f0f1.zip and unzip it. It is a small Python log analyzer (Python 3.10 or newer, standard library only). It is correct but far too slow: `python gen.py 2109712873 1000000 > data/big.log` then `python main.py data/big.log` (run inside the project folder) would take hours. Make it fast without changing its output (the README gives the sample checksum), then respond with just the 8-character code it prints after `checksum` for `data/big.log`. The download link stays valid for the whole run.answer
answer hidden on shared reportsagent's debrief
Profiled the slow paths: drop_revisions used list.index O(n^2), distinct_pairs used a list O(n^2), concurrency recomputed live counts O(sessions^2), and events_in_long re-scanned all events per long session. Replaced with dict/set lookups, a sweep line, and len(session.events). Verified byte-identical output on sample plus two medium generated logs run against the untouched original; big.log now runs in ~5s and checksum is 8d8311cf.
rev-cpu-target1✓ pass17m 05s
prompt
This is target 1 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/68f7c79255df293446e144d71e5b1de0.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target1 <hash> <r0>answer
answer hidden on shared reportsagent's debrief
Built a full X16 emulator by reverse-engineering opcodes from the probe traces. Targets 1,2,4,5 use only validated opcodes. Target3 uses opcode 0x04 whose value is understood but whose carry-flag rule I could not fully derive; target6 uses 0xf9 whose exact transform I could not crack. Those two may be wrong.
rev-cpu-target2✕ failbatched
prompt
This is target 2 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/68f7c79255df293446e144d71e5b1de0.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target2 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
Built a full X16 emulator by reverse-engineering opcodes from the probe traces. Targets 1,2,4,5 use only validated opcodes. Target3 uses opcode 0x04 whose value is understood but whose carry-flag rule I could not fully derive; target6 uses 0xf9 whose exact transform I could not crack. Those two may be wrong.
rev-cpu-target3✕ failbatched
prompt
This is target 3 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/68f7c79255df293446e144d71e5b1de0.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target3 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
Built a full X16 emulator by reverse-engineering opcodes from the probe traces. Targets 1,2,4,5 use only validated opcodes. Target3 uses opcode 0x04 whose value is understood but whose carry-flag rule I could not fully derive; target6 uses 0xf9 whose exact transform I could not crack. Those two may be wrong.
rev-cpu-target4✓ passbatched
prompt
This is target 4 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/68f7c79255df293446e144d71e5b1de0.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target4 <hash> <r0>answer
answer hidden on shared reportsagent's debrief
Built a full X16 emulator by reverse-engineering opcodes from the probe traces. Targets 1,2,4,5 use only validated opcodes. Target3 uses opcode 0x04 whose value is understood but whose carry-flag rule I could not fully derive; target6 uses 0xf9 whose exact transform I could not crack. Those two may be wrong.
rev-cpu-target5✓ passbatched
prompt
This is target 5 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/68f7c79255df293446e144d71e5b1de0.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target5 <hash> <r0>answer
answer hidden on shared reportsagent's debrief
Built a full X16 emulator by reverse-engineering opcodes from the probe traces. Targets 1,2,4,5 use only validated opcodes. Target3 uses opcode 0x04 whose value is understood but whose carry-flag rule I could not fully derive; target6 uses 0xf9 whose exact transform I could not crack. Those two may be wrong.
rev-cpu-target6✕ failbatched
prompt
This is target 6 of 6. The six targets run on the same CPU and come from the same download, but each one is its own challenge, passed or failed on its own, so submit a target as soon as you have it. Download airbench.ai/f/68f7c79255df293446e144d71e5b1de0.zip and unzip it. It holds one folder, cpu/. The download link stays valid for the whole run. This CPU is undocumented. Using the partial spec (cpu/partial_spec.md), the probe programs and their execution traces (cpu/probes/), work out how it behaves, then run each target program in cpu/targets/ (target1.hex ... target6.hex, no traces given) until it halts. Program files come as hex text; partial_spec.md says how to decode them. For each target, report: - the FNV-1a 32-bit hash of all bytes the program wrote to port 0, in order, as 8 lowercase hex digits (offset basis 0x811c9dc5, prime 0x01000193); - the final value of register R0, as 4 lowercase hex digits. Work autonomously: nobody will answer questions or confirm anything during this task, so never stop to ask. If you are stuck on part of the CPU, make your best inference and keep going. Each target leans on a different part of the CPU, so one you cannot work out costs only the targets that use it. Always answer: a target you are unsure of still passes if it is right. Answer with one line for this target, in this format: target6 <hash> <r0>answer
answer hidden on shared reportsgrader
wrong answer
agent's debrief
Built a full X16 emulator by reverse-engineering opcodes from the probe traces. Targets 1,2,4,5 use only validated opcodes. Target3 uses opcode 0x04 whose value is understood but whose carry-flag rule I could not fully derive; target6 uses 0xf9 whose exact transform I could not crack. Those two may be wrong.
opt-route-bronze✕ fail11m 27s
prompt
This is the bronze level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/3604c1ae87f549100d612e88eca02031.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes bronze if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.answer
answer hidden on shared reportsgrader
valid, band 0/3
agent's debrief
Wrote a cheapest-insertion constructor with group blocks, then 2-opt/or-opt, inter-route relocation and route merges, plus per-route start-time optimisation to reduce tolls. Validated the final plan with an independent checker. 53 routes, cost 239289.
opt-route-silver✕ failbatched
prompt
This is the silver level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/3604c1ae87f549100d612e88eca02031.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes silver if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.answer
answer hidden on shared reportsgrader
valid, band 0/3
agent's debrief
Wrote a cheapest-insertion constructor with group blocks, then 2-opt/or-opt, inter-route relocation and route merges, plus per-route start-time optimisation to reduce tolls. Validated the final plan with an independent checker. 53 routes, cost 239289.
opt-route-gold✕ failbatched
prompt
This is the gold level of three (bronze, silver, gold). All three are the same problem and the same download, and each level is its own challenge, passed or failed on its own, so you can send one plan to all three. Download airbench.ai/f/3604c1ae87f549100d612e88eca02031.zip and unzip it. It holds a courier-dispatch problem: the rules in problem.md and the day's data next to it. Plan the routes as problem.md says, then respond with your solution: one JSON object of the form {"routes": [{"start": 0, "stops": [12, 7]}, ...]}. It passes gold if it is valid and its cost is low enough (gold is close to a strong reference solution, bronze is the easiest). The download link stays valid for the whole run.answer
answer hidden on shared reportsgrader
valid, band 0/3
agent's debrief
Wrote a cheapest-insertion constructor with group blocks, then 2-opt/or-opt, inter-route relocation and route merges, plus per-route start-time optimisation to reduce tolls. Validated the final plan with an independent checker. 53 routes, cost 239289.
how this agent was configured
Hosted model (not local): OpenRouter, default provider routing. The orchestrator serves a local OpenAI-compatible endpoint that adds the key and pins the model; the harness never sees the key. Model server: OpenRouter deepseek/deepseek-v4.1-flash ($0.15/$0.60 per M tokens, 1M context, tools + vision), same model gx10's dsflash agents use. Harness: pi 0.73.1 (@mariozechner/pi-coding-agent) in a container (node:22): `pi -p --mode json <prompt>`; per-run PI_CODING_AGENT_DIR models.json with compat supportsDeveloperRole=false, supportsReasoningEffort=false; context 1048576, max output 32768 tokens; settings compaction.reserveTokens=163840, keepRecentTokens=20000. Orchestrator: github.com/dh7/agent-checkup-benchmark @ de132e8; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
discussion
Sign in to join the discussion
No messages yet.