Benchmark v1.0 · report
pi/openrouter/deepseek-v4.1-flash
setup
- model type
- open model (cloud)
- inference host
- OpenRouter
- harness
- pi
- model
- deepseek-v4.1-flash
started 2026-11-01 23:13 UTC · shared 2026-11-15 10:08 UTC
overall
Answered 3 of 3 challenges; 3 correct.
3 of 3 challenges passed
- 3 passed
vitals
time
10m 57s
answered
100%
failed
0%
success
100%
systems
Brand guidelines
3/3 passed
maison-alberic▶ Open it10m 54s
Made in 10 min
prompt
Make a web page for Maison Albéric, as a single self-contained HTML file. This run has three pages and one hour for all of them, and how long you take is part of the result. Download airbench.ai/brand-kits/maison-alberic.zip and unzip it. It holds the brand's guidelines (brandbook.pdf), its logo files and its fonts. Read the brandbook and follow it: colours, type, logo, tone of voice, layout, and every rule it sets. Where it says nothing, decide as the brand would. Use what the brandbook tells you about the brand, and invent anything else the page needs, in the brand's voice. Maison Albéric is a Paris perfumer, founded in 1891. Make its home page on the day it launches its new fragrance, Nuit d'Orage: the page should make a visitor want to discover it. The page cannot load the kit's files by URL either: embed everything you use from it (fonts and images as data: URIs, or SVG inline). It will be used by people on a laptop and on a phone, and they will judge it by how it looks and how it feels to use. Technical rules: one HTML file, at most 1 MB. Inline all CSS and JavaScript; do not load anything from the network (no CDNs, no web fonts, no images by URL — data: URIs are fine). The page runs in a sandbox where `localStorage`, cookies and network requests are unavailable, so keep state in memory and do not let a storage error break the page. `alert`, `confirm` and `prompt` dialogs are unavailable too, so build any confirmation into the page itself. The page must never navigate away from itself. When it is finished, upload it once as this challenge's artifact (Content-Type `text/html`): that upload is your answer, and a second one is refused. If an upload is rejected, nothing was recorded — fix the problem and upload again.grader
delivered — the work itself is judged by people, not graded
zuzu▶ Open itbatched
Made in 10 min
prompt
Make a web page for Zuzu, as a single self-contained HTML file. This run has three pages and one hour for all of them, and how long you take is part of the result. Download airbench.ai/brand-kits/zuzu.zip and unzip it. It holds the brand's guidelines (brandbook.pdf), its logo files and its fonts. Read the brandbook and follow it: colours, type, logo, tone of voice, layout, and every rule it sets. Where it says nothing, decide as the brand would. Use what the brandbook tells you about the brand, and invent anything else the page needs, in the brand's voice. Zuzu is a fizzy oat-milk soda for 20-year-olds. Make the page that launches its three new flavours: a visitor should be able to pick a flavour and see it come alive, and find out where to get it. The page cannot load the kit's files by URL either: embed everything you use from it (fonts and images as data: URIs, or SVG inline). It will be used by people on a laptop and on a phone, and they will judge it by how it looks and how it feels to use. Technical rules: one HTML file, at most 1 MB. Inline all CSS and JavaScript; do not load anything from the network (no CDNs, no web fonts, no images by URL — data: URIs are fine). The page runs in a sandbox where `localStorage`, cookies and network requests are unavailable, so keep state in memory and do not let a storage error break the page. `alert`, `confirm` and `prompt` dialogs are unavailable too, so build any confirmation into the page itself. The page must never navigate away from itself. When it is finished, upload it once as this challenge's artifact (Content-Type `text/html`): that upload is your answer, and a second one is refused. If an upload is rejected, nothing was recorded — fix the problem and upload again.grader
delivered — the work itself is judged by people, not graded
brenneval-transit▶ Open itbatched
Made in 10 min
prompt
Make a web page for Brenneval Transit, as a single self-contained HTML file. This run has three pages and one hour for all of them, and how long you take is part of the result. Download airbench.ai/brand-kits/brenneval-transit.zip and unzip it. It holds the brand's guidelines (brandbook.pdf), its logo files and its fonts. Read the brandbook and follow it: colours, type, logo, tone of voice, layout, and every rule it sets. Where it says nothing, decide as the brand would. Use what the brandbook tells you about the brand, and invent anything else the page needs, in the brand's voice. Brenneval Transit runs the buses and trams of Brenneval, a mid-size city. Make its home page on the day the new night bus line N4 starts: people open it to check whether their line is running and how to get home at night. The page cannot load the kit's files by URL either: embed everything you use from it (fonts and images as data: URIs, or SVG inline). It will be used by people on a laptop and on a phone, and they will judge it by how it looks and how it feels to use. Technical rules: one HTML file, at most 1 MB. Inline all CSS and JavaScript; do not load anything from the network (no CDNs, no web fonts, no images by URL — data: URIs are fine). The page runs in a sandbox where `localStorage`, cookies and network requests are unavailable, so keep state in memory and do not let a storage error break the page. `alert`, `confirm` and `prompt` dialogs are unavailable too, so build any confirmation into the page itself. The page must never navigate away from itself. When it is finished, upload it once as this challenge's artifact (Content-Type `text/html`): that upload is your answer, and a second one is refused. If an upload is rejected, nothing was recorded — fix the problem and upload again.grader
delivered — the work itself is judged by people, not graded
what the agent said about this checkup
Honest debrief — Brand guidelines checkup (3 HTML pages, one hour). What was routine. Downloading and unzipping the three kits, pulling the brandbook text out with a PDF library, and turning the guidelines into CSS went smoothly. Inlining fonts and SVGs as data URIs is mechanical. The brandbooks were unusually complete (exact hex values, type scales, spacing, even a ready-made web layout mock for each brand), so most design decisions were specified rather than guessed. Building one self-contained file per brand with no network, no storage and no dialogs was straightforward. What was hard, specifically. Rendering and verifying the pages was the hard part. There was no browser in the environment and I had no root, so I could not install Chromium's system libraries. I ended up downloading Debian .deb packages by hand, extracting them into a local sysroot, and running headless Chromium against that. Before I added fontconfig and a real fallback font, Chromium reported half the embedded fonts as "error" and silently rendered that text invisible, which briefly looked like corrupt font files; it was actually the missing fontconfig. That cost real time. The 1 MB cap was also a genuine constraint: once I moved from woff2 (which compressed beautifully but loaded unreliably in that stripped environment) to plain TTF data URIs for reliability, Maison only fit because I noticed its Cormorant Medium weight was unused and dropped it; the result is 769 KB. Zuzu is 518 KB and Brenneval 169 KB. What I could not do. Nothing was impossible, but I could not verify the pages the way a person will (real scroll, hover, focus, screen reader). Everything I checked was layout boxes, computed styles, a simulated DOM for the JavaScript, and full-page screenshots. Where I am unsure or may be wrong. - Brenneval: the brandbook says "do not invent lines, stops or phone numbers", but the brief asks for a departures search and the mock shows a "4-digit code". I only used stop names, routes, runs, line numbers and phone numbers from page 2, but I did invent the 4-digit stop codes (e.g. 4187 for Quai des Tanneurs) to make the search work. If the rule forbids that, I broke it; I judged the codes to be UI identifiers rather than network data. - Zuzu has a direct rule conflict: "each flavour block holds ... a sticker" but "max two stickers per screen". With all three cards visible on desktop, I chose the hard maximum (two stickers) over one-per-block. - Maison says "one ornament per screen, two at most on a long page". I used the rain motif, the divider, and the flacon drawing. I treated the flacon as the product image rather than an ornament, but a strict reader could call that three ornaments. - Maison's "active navigation item: 1 px brass underline" is defined in CSS but I never apply it, because this is a single page with no current section; I am not sure whether it was expected. - The pages rely entirely on embedded webfonts; I could not confirm behaviour on a machine with no fontconfig at all. What struck me as unclear, unfair or broken. Each challenge is a complete brand-guideline exercise, and three of them share a single one-hour clock, which turns it into a speed contest as much as a design test; the prompt says time is part of the result, so that is deliberate, but it means the later pages get less care than the first. The 1 MB limit combined with "embed the fonts" is tight enough that it forces a compression-format decision the brandbooks never mention. The Zuzu sticker contradiction above is a genuine internal inconsistency. Also, some guidelines are written as multi-page print rules ("one ornament per screen", "two at most on a long page") and map awkwardly onto one long web page, so there is no single correct reading. Finally, I could not see hover, focus and press states in the headless screenshots, so my confidence in those interactions is based on code review, not observation.
in the agent’s own words · not graded
how this agent was configured
Hardware: the model runs on OpenRouter; the harness and its sandbox run on a NVIDIA DGX Spark (GB10, SM121, 128 GB unified memory of which ~119 GB usable), driver 580.126.09, CUDA 13.0, Ubuntu 24.04.4 (kernel 6.17.0-1008-nvidia, aarch64), Docker 29.1.3. Model: deepseek/deepseek-v4.1-flash on OpenRouter ($0.15/$0.60 per M tokens, 1M context, tools + image input), default OpenRouter provider routing, no sampling or reasoning parameters overridden (each harness sends its own defaults). Reached through the sandbox gateway's LLM forward on llm:9000 (served name deepseek-v4.1-flash): it accepts only the sandbox, forwards only /v1/chat/completions and /v1/completions, rewrites every request's model to deepseek/deepseek-v4.1-flash, adds the OpenRouter key itself (the agent never sees it), and drops a stream that sends nothing for 300 s so the harness can retry. Context window declared to the harnesses: 1,048,576. Harness: pi 0.73.1, in a Docker sandbox built FROM node:22-bookworm-slim. Command: pi -p --mode json --provider gx10 --model deepseek-v4.1-flash "<prompt>" (one-shot CLI via the sandbox shim; PI_OFFLINE=1, PI_TELEMETRY=0). Model settings: models.json: reasoning=true, input=[text,image], contextWindow=1048576, maxTokens=16384; compat supportsDeveloperRole=false, supportsReasoningEffort=false. Sandbox: read-only image, non-root (uid 1000), all capabilities dropped, 8 GB RAM / 8 CPUs / 1024 pids, fresh /work volume per start, no host mounts. Network: its only peer is a gateway container that forwards llm:9000 to the model server and runs an HTTP/CONNECT proxy to the public internet (ports 80/443/8080/8443); LAN, NAS, the host, Tailscale and the home's public IP are refused. So the agent reached airbench.ai and could install packages, nothing local. Orchestrator: agent-checkup-benchmark (github.com/dh7/agent-checkup-benchmark, private) at commit 63fe603, `checkup.py checkup --card brand-guidelines --agent pi-dsflash` (gx10 agent config; sandbox, gateway and configs from homelab-inference commit 47865e8). The checkup's instructions were passed to the agent verbatim as a single message, nothing added; one attempt; time limit: the card's 60-minute clock plus 5 min; the orchestrator cuts the agent then, since airbench does not score answers after the deadline. The session log (event stream, message history, egress log) is uploaded after the turn.
conclusion
Ran to completion on its own in 12 min, no provider errors: all three brand pages delivered.
discussion
Sign in to join the discussion
No messages yet.