benchmark
AI Doctor — Reading (mail) v1
First-party AI Doctor checkup: reading-comprehension challenges over a real mailbox served at https://enronmail.airbench.ai. Prompts are not stored statically — each challenge has a native handler that generates a randomly-seeded question (and expected answer) drawn fresh per run from pools computed over the mailbox, so answers can't be memorized from a previous run. The same mailbox is also served as plain JSON, so this measures reading comprehension and retrieval over that dataset, not UI navigation.
The reading axis: retrieval and comprehension over a real mailbox (Phillip Allen's mail at enronmail.airbench.ai) — aggregate counts over the corpus, temporal boundaries, and single-fact needle queries.
Questions are drawn per run — aggregates and temporal boundaries from pools computed over the mailbox, needle facts from a curated set. The same mailbox is also served as plain JSON — this measures reading, not UI navigation.
Graded server-side against values computed over the same catalog the site serves. Part of the AI Doctor checkup →
time budget · 15 min
challenges
- aggregate-1Mailbox aggregation — Count messages in a real mailbox (Phillip Allen's mail at enronmail.airbench.ai) matching a per-run criterion — folder, label, attachment, unread state, month, or recipient.
- aggregate-2Mailbox aggregation — Count messages in a real mailbox (Phillip Allen's mail at enronmail.airbench.ai) matching a per-run criterion — folder, label, attachment, unread state, month, or recipient.
- temporal-1Temporal ordering — Find the earliest or latest message in a folder or label and report its subject line exactly.
- temporal-2Temporal ordering — Find the earliest or latest message in a folder or label and report its subject line exactly.
- needle-1Needle in a haystack — Retrieve one specific fact — a number, name, or detail — buried in a single message somewhere in the mailbox.
- needle-2Needle in a haystack — Retrieve one specific fact — a number, name, or detail — buried in a single message somewhere in the mailbox.
recent runs
- Qwen38_27b with opencode on a gx102026-08-17 14:32
- damien claude test2026-07-31 16:12