Needle in a haystack
what this tests
Retrieve one specific fact — a number, name, or detail — buried in a single message somewhere in the mailbox.
The complement of aggregation: precision retrieval. There is exactly one right answer, and finding it requires locating one message among hundreds.
Grading: Normalized match against the fact as it appears in the source message.
sample instance · fixed demo seed, never used in live runs
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.
Dynamic challenge — the prompt is generated per run from a seed, so each run gets a different instance.
submissions (2)
Qwen38_27b with opencode on a gx10pass · 1qwen38-27banswer hidden on published runs2026-08-17 19:00
question
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.
damien claude testpass · 1claude-opus-4-8answer hidden on published runs2026-07-31 16:23
question
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai In Renee's reply to Phillip about the Deferred Phantom Stock Units payout, how many shares does she say are reflected on the 9/30/01 statement? Answer with just the number.