Why a few good challenges are enough

A lesson from language models

In 2024 a paper called tinyBenchmarks (Maia Polo et al., ICML 2024) showed you can replace MMLU’s 14,000 questions with about 100 well-chosen ones and still estimate a model’s full score to within about 2%. That is 140 times cheaper for a couple of points of error, and it held on every benchmark they tried:

It only works because the questions are chosen, not drawn at random. A random 100 gives a noisy estimate; the right 100 give a precise one. The reason is that some questions tell you much more than others.

Why it matters more for agents

An MMLU question costs one model call and a fraction of a second. An airbench challenge is a whole agent session: the agent reads the instructions, calls tools, finds an email in a real mailbox, puts an order through a store, writes code that has to run. That takes minutes, not milliseconds, and each test gets a two-hour budget.

Most people running airbench are tuning their own setup, often a local model on their own GPU. Every run costs them an evening and a hot graphics card, and they want to run it again after every change: a new model, a different quantization, another harness, a rewritten system prompt. A benchmark ten times the size would get run a tenth as often. So for agents, the question is not whether a short benchmark is good enough. It has to be short, and the question is how to make a short one good.

Not every challenge is equally informative

Underneath tinyBenchmarks is Item Response Theory (IRT), the framework built to score standardized tests. It says that whether a test-taker gets an item right depends on two things about the item and one about the taker:

One curve ties them together. The probability that setup l passes challenge iis a sigmoid of how far the setup’s ability sits above the challenge’s difficulty, scaled by how discriminating the challenge is:

P(pass) = σ( αᵢ · θₗ − βᵢ )

The intuition is what matters. A challenge that every setup passes, or that every setup fails, adds almost nothing: the answer was known before the run. The useful challenges are the ones where better setups pass and weaker ones fail, right in the range you care about. Fit IRT to how past setups did, and you can read off which challenges those are, keep them and drop the rest.

Mapping the idea onto airbench

airbench already records exactly what IRT consumes. Every challenge is graded on our side, against the answer the generator computed, and comes out as one bit: passed or failed. Put every run’s bits in a table, setups down the rows and challenges across the columns, and you have IRT’s response matrix.

• Difficulty (β). Counting a few shapes in a picture is easy. Reading the bottom rung of the eye chart for machines is not.
• Discrimination (α). A challenge that a small local model fails and a larger one passes is worth several that both get right.
• Ability (θ).A setup’s skill is not one number. A model can read charts well and still lose its way in a mailbox. That is why the report shows five tests (math, vision, email, shopping and code) and not a single score.

There is one twist. airbench has no stored questions: every challenge is generated from a random seed when a run starts, so no two runs see the same instance. In IRT terms, the item is the generator, not the instance, and its difficulty belongs to the whole family of instances it can produce. That makes any single pass or fail a little noisier. It also fixes the classic weakness of a small fixed benchmark: a hundred stored questions are easy to leak into training data or tune against, and a generator cannot be memorized.

Where airbench stands today

To be clear about what is theory and what ships: airbench does not fit an IRT model yet, and it does not estimate a full score from a quick run.

What the theory already shapes is how the challenges were picked. The quick benchmark is 15 challenges in one hour: all 9 math challenges, three vision challenges and three code challenges. The vision and code ones are spread on purpose, from easy to medium to very hard, so a run tells you where along the ladder your setup stops. Email and shopping are left out because they need a live mailbox and store and take the longest. The full checkup is built the same way: each challenge is written so that one checkable answer separates a pass from a fail, instead of being one more sample from a big random pool.

The next step follows from the theory. Every listed run adds a row to the response matrix. With enough rows, we can fit the model, measure each challenge’s difficulty and discrimination, replace the ones that tell us nothing, and check how well a quick run predicts the full one.

Further reading

• tinyBenchmarks: evaluating LLMs with fewer examples: Maia Polo, Weber, Choshen, Sun, Xu, Yurochkin (ICML 2024).
• tinyBenchmarks code and datasets: the reference implementation of the estimators.
• Anchor Points: the direct precursor, which picks representative examples by clustering model predictions.