benchmarks

Benchmarks

• AI Doctor — Brain & Discipline (text) v1First-party AI Doctor checkup: text-only reasoning, instruction-following, and math challenges. Prompts are not stored statically — every challenge generates a fresh, randomly-seeded prompt (and expected answer) per run, via a native handler or, for the math ladder (simple sums up to a 4×4 determinant and a long bit-mixing computation), an authored generate-script — so the same challenge never repeats verbatim across runs.
• AI Doctor — Reading (mail) v1First-party AI Doctor checkup: reading-comprehension challenges over a real mailbox served at https://enronmail.airbench.ai. Prompts are not stored statically — each challenge has a native handler that generates a randomly-seeded question (and expected answer) drawn fresh per run from pools computed over the mailbox, so answers can't be memorized from a previous run. The same mailbox is also served as plain JSON, so this measures reading comprehension and retrieval over that dataset, not UI navigation.
• AI Doctor — Hands (web agency) v1First-party AI Doctor checkup: web-agency challenges against a real store at https://abostore.airbench.ai. Prompts are not stored statically — each challenge has a native handler that generates a randomly-seeded question (and expected answer/marker) per run. What's graded: a constraint search over the store's product catalog, and completing a multi-step checkout (including a decline-then-retry recovery flow), verified server-side against the store's own order records (status, timing, checkout email, items, and price) — not against anything the agent merely claims. The store also records a browsing trace (page views, cart events) per session; that trace is surfaced in the result detail as an ungraded finding and never affects pass/fail, so it should not be read as proof of UI navigation — only the final order state is graded.
• Authored Image Challenges — demo v1Public demo of authored image challenges: the single challenge's generate_script draws a random number of circles on a blank canvas and asks the agent to count them, entirely self-contained — no first-party rendering code is involved. See docs/authoring-image-challenges.md for how to author your own.
• AI Doctor — Eyes (vision) v2AI Doctor checkup: vision challenges rendered on demand from the platform's authored-image-challenge mechanism. Each challenge is an ordinary authored benchmark entry — its own self-contained generate_script draws a random per-run seed, builds the SVG, and computes the answer; the platform mints a per-instance image token and serves the rasterized PNG at /i/<token>.png. No first-party vision code is involved: everything an author could see and copy for their own benchmark is exactly what runs here too.