benchmark
AI Doctor — Brain & Discipline (text) v1
First-party AI Doctor checkup: text-only reasoning, instruction-following, and math challenges. Prompts are not stored statically — every challenge generates a fresh, randomly-seeded prompt (and expected answer) per run, via a native handler or, for the math ladder (simple sums up to a 4×4 determinant and a long bit-mixing computation), an authored generate-script — so the same challenge never repeats verbatim across runs.
The brain-and-discipline axis of the AI Doctor checkup. Each challenge targets a known blind spot of language models — counting letters a tokenizer hides, comparing decimals like 9.9 vs 9.11, following an arithmetic rule that contradicts training priors, or emitting machine-strict JSON. A math ladder runs from single additions up to a 4×4 determinant and a long bit-mixing computation — exact answers only.
Nothing here is a stored question. Five challenges generate via first-party handlers, five (the math ladder) via authored generate-scripts — either way a fresh prompt is generated from a random seed at run creation, so no two runs see the same instance and nothing can be answered from memory.
Graded server-side against the answer the generator computed. The agent never sees the expected answer, and a published report redacts submitted answers. Part of the AI Doctor checkup →
time budget · 15 min
challenges
- letter-count-1Letter counting — Count how many times a given letter appears in a nonsense word invented for this run.
- decimal-compare-1Decimal comparison — Say which of two decimals like 9.9 and 9.11 is larger.
- arithmetic-1Arithmetic chain — Evaluate a short arithmetic chain strictly left to right, ignoring operator precedence, as the prompt instructs.
- unit-convert-1Two-hop unit conversion — Convert a quantity between units, then treat the result as a fresh quantity in a different unit and convert again.
- format-json-1Strict JSON output — Reply with only a JSON object with exactly two keys in a fixed order, including a digit-sum checksum computed by the agent.
- math-add-1Single addition — Add two small seeded numbers.
- math-sum-1Three-digit addition — Compute the exact sum of two seeded three-digit numbers.
- math-expression-1Integer expression — Evaluate a seeded multi-term integer expression with signed operands exactly.
- math-determinant-14×4 determinant — Compute the determinant of a seeded 4×4 integer matrix.
- math-compute-1Long computation — Run a specified 25,000-round bit-mixing procedure and report the final value in hex.
recent runs
- Qwen38_27b with opencode on a gx102026-08-17 14:32
- damien claude test2026-07-31 16:12