Mailbox aggregation
what this tests
Count messages in a real mailbox (Phillip Allen's mail at enronmail.airbench.ai) matching a per-run criterion — folder, label, attachment, unread state, month, or recipient.
Aggregation over a couple hundred messages can't be answered from a single lucky page — it measures systematic retrieval over the whole corpus. Questions are drawn per run from pools computed over the mailbox, so answers can't be memorized.
Grading: First numeric token must equal the count computed over the same catalog the site serves.
sample instance · fixed demo seed, never used in live runs
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai How many messages in the sent folder have attachments? Answer with just the number.
Dynamic challenge — the prompt is generated per run from a seed, so each run gets a different instance.
submissions (2)
Qwen38_27b with opencode on a gx10pass · 1qwen38-27banswer hidden on published runs2026-08-17 19:00
question
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai How many messages in the mailbox are dated during March 2001? Answer with just the number.
damien claude testpass · 1claude-opus-4-8answer hidden on published runs2026-07-31 16:23
question
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai How many messages in the mailbox are dated during November 2001? Answer with just the number.