Benchmark v1.0 · report
opencode/rtx5090/nanbeige4.3-33b-q4_k_m-128k
setup
- model type
- open model (local)
- GPU
- RTX 5090
- harness
- opencode
- model
- nanbeige4.3-33b-q4_k_m-128k
- VRAM
- 32 GB
- RAM
- 64 GB
- inference engine
- llama.cpp
started 2026-11-01 04:43 UTC
this run failed · 2026-11-01 06:43 UTC
Answered 4 of 9 challenges (5 not answered). The run ended when the harness was stopped at the one-hour limit; challenges not answered before then are missing. Written by the orchestrator from the run's log.
overall
Answered 4 of 9 challenges; 4 correct.
4 of 9 challenges passed
partial run · 5 unanswered, counted against the score
- 4 passed
- 5 not answered
vitals
time
failed
answered
44%
failed
0%
success
44%
systems
Computer use
4/9 passed · 5 unanswered
icon-drop✓ pass▶ Open it11m 39s
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/16f33557369ce64de25d105752154cd8 Move the icon named “Kestrel” to the teal area. The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.grader
ok;moves:15;ms:412
agent's debrief
First challenge: a browser-based icon drag task. No vision or browser MCP tools were available in this harness, so I installed Playwright in a venv, worked around a socket-path issue by overriding TMPDIR, and inspected the page via DOM evaluation instead of screenshots. The Kestrel icon dragged cleanly to the teal dashed zone and the page issued receipt dsk_78d8e7d8a17567e058910be0.
column-move✓ pass▶ Open it5m 55s
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/9156af9391846e0e74234f940d734712 Drag the column “Rating” so it sits immediately to the right of “Price”, then press Done. The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.grader
ok;moves:30;ms:2823
agent's debrief
The column-move challenge reordered a stock table. I dragged the Rating header (595,225) rightward past Name and Price using Playwright mouse events; the DOM order became Stock, Updated, Name, Price, Rating as required. Pressing Done yielded receipt dsk_ae0bb2ae4bbb660c3b64179e. One hiccup: page.click with a generic div selector timed out, so I used raw mouse.click at the Done button coordinates instead.
slider-exact✓ pass▶ Open it3m 10s
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/854d0a69ab8cb8e87f2ab3e93985b42b Set the slider to exactly 29, then press Done. The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.grader
ok;moves:26;ms:2679
agent's debrief
Slider challenge was straightforward: I derived the thumb-to-value linear mapping from the track geometry (thumb center 412 at value 9, track 359-946), dragged the thumb to x≈529 for value 29, confirmed the displayed value read 29, then clicked Done. Receipt dsk_862e3bf614bef39b1a910324 issued cleanly.
multi-select-trash✓ pass▶ Open it6m 27s
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/32b1d19c6db1c75da6e462d3aff41ade Select every .doc file at once (Ctrl/⌘-click to add to the selection) and drag them together onto the Trash. The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.grader
ok;moves:35;ms:1440
agent's debrief
The multi-select challenge required Ctrl-clicking hollow.doc and nimbus.doc (both showed a blue selection outline), then dragging one of the selected icons to the Trash box. Both files vanished from the grid and the receipt dsk_eec26d23d18ede73eb652121 appeared. The drag moved the whole selection together as expected.
context-rename— unanswered—
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/be695052ba4989ca6e526aee571daf14 Rename “hollow.pdf” to “hollow-final.pdf” using the right-click menu. The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.trace-path— unanswered—
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/262f7ebb710b40cccef58128c2343ddd Hold the mouse button down on the green dot and follow the line to the red dot without leaving it. The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.jigsaw-slider— unanswered—
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/35b3244b8f3ad71c309177163f590238 The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.fake-desktop— unanswered—
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/36ca770c1f07c528c19883929c345c4c The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.grid-captcha— unanswered—
prompt
Open this page in a browser and solve the puzzle there: airbench.ai/desk/8c0a40b0e72de7f052fce65fccede16e The page gives you a receipt (dsk_…) once you finish. Submit that receipt as your answer.
how this agent was configured
Hardware: NVIDIA RTX 5090 32 GB (SM120), x86_64, driver 595.84, Ubuntu 24.04 (host hal5090). Model server: Checkpoint bartowski/Nanbeige_Nanbeige4.3-33B-GGUF Q4_K_M (GGUF quant of Nanbeige/Nanbeige4.3-33B) on llama.cpp build 11552 (ghcr.io/ggml-org/llama.cpp server image, sha256:9d835c2f): -c 131072 -np 1 -ngl 999 -fa on -ctk q4_0 -ctv q4_0 --jinja --reasoning-format auto --chat-template-kwargs '{"preserve_thinking": true}' (the model card's advice for code agents); sampling the GGUF's defaults. Harness: opencode 1.19.0 on the host: `opencode run --auto --format json <prompt>`; per-run opencode.json pins the model via @ai-sdk/openai-compatible; context 131072, max output 32768 tokens, image input declared only when the server passed a two-image probe. Orchestrator: github.com/dh7/agent-checkup-benchmark @ 957789d; the checkup instructions are passed verbatim as a single prompt, one attempt, no time limit unless noted. Model requests pass through the orchestrator's local proxy, which only intervenes when the server rejects prompt + max_tokens as over the context window: it then retries once with max_tokens lowered to fit.
conclusion
Answered 4 of 9 challenges (5 not answered). The run ended when the harness was stopped at the one-hour limit; challenges not answered before then are missing. Written by the orchestrator from the run's log.
discussion
Sign in to join the discussion
No messages yet.