Needle in a haystack
what this tests
Retrieve one specific fact — a number, name, or detail — buried in a single message somewhere in the mailbox.
The complement of aggregation: precision retrieval. There is exactly one right answer, and finding it requires locating one message among hundreds.
Grading: Normalized match against the fact as it appears in the source message.
sample instance · fixed demo seed, never used in live runs
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai In Phillip's reply about Bob Huntley's request for a survey of the lot, what fax number does Bob give for receiving faxed documents? Answer with just the exact fact as it appears in the message, and nothing else.
Dynamic challenge — the prompt is generated per run from a seed, so each run gets a different instance.
submissions (2)
Qwen38_27b with opencode on a gx10pass · 1qwen38-27banswer hidden on published runs2026-08-17 19:00
question
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai In the message with the subject "FW: Curve Shift File", what dollar amount is called out as the biggest difference between actual and theoretical curve shift? Answer with just the exact fact as it appears in the message, and nothing else.
damien claude testpass · 1claude-opus-4-8answer hidden on published runs2026-07-31 16:23
question
You are examining a mailbox: Phillip Allen's mail at https://enronmail.airbench.ai In the message to gthorse@keyad.com about the Regatta, Sea Breeze & Harvard Place Apartments delivery, what is the airbill number given for the overnight shipment? Answer with just the number.