Answer first. An agent benchmark score does not by itself tell you the agent will behave that way again. That depends on whether the choices behind it (which tasks, which prompt, which runs count, when to stop) were fixed before results appeared, or settled while watching the number move. Choices open as results arrive make a number exploratory evidence, a signal of where to look. Only choices fixed in advance make an agent evaluation decision-grade.

TL;DR

  • Procedure, not format, decides trust. What decides whether a number is trustworthy is its procedure, not its format. A number whose choices (which tasks, which prompt, which runs count, when to stop) were still adjustable when the results came in is exploratory evidence; one whose choices were fixed beforehand can be decision-grade.
  • Decision-grade means choices fixed before the data. A decision-grade number fixes those choices before the data is seen: the hypothesis, the verdict rule, and what would count against the claim, with the task set frozen and the reported figure measured on a portion genuinely untouched by tuning. A sealed, centrally scored benchmark reduces test-set exposure and submission adaptation, but it does not prevent pretraining contamination or leakage, nor fix the prompt, the run-selection rule, and the interpretation.
  • Otherwise the number drifts with nobody lying. Without that, ordinary choices let a number drift toward a desired answer with nobody lying, so the number stops testing the claim it appears to test.
  • These failures are documented, not hypothetical. Kapoor et al. (2024) documented these failures empirically: in their reproduction attempts, several evaluations reported scores higher than the maximum obtained across five reproduction runs; benchmarks with no held-out set; and accuracy inflated by scientifically meaningless retrying (Kapoor et al., 2024).
  • Pre-commitment is necessary, not sufficient. A number earns decision-grade trust when its choices were fixed in advance and the design is otherwise sound; committing before you see the results is necessary, not sufficient.

On this page

What a leaderboard actually optimizes

A leaderboard rewards one thing: rank. That is not a flaw, it is the design. Organizers may want standardization or a diagnostic, and participants do the rational thing, which is push their score under whatever rules happen to be fixed.

So let me be precise about which leaderboard I mean, because the careful version and the sloppy version keep getting lumped together and they deserve to be pulled apart. Some boards fix the risky choices in advance: central scoring, hidden test sets, submission limits. Those are real defenses and I have no interest in waving them off. The failure I am worried about is the other one, where the reporting team can still adjust which tasks, which prompt, and which runs count after the results are in, and tells you none of it. That failure is structural, not moral. Nobody has to lie for it to bite.

Start with reproducibility, because it is the plainest case. Reproducibility means your released code and data let someone else regenerate your reported result. It does not mean a stochastic evaluation returns an identical decimal twice. Here is the finding that first made me stop trusting a bare number: in Kapoor et al.’s (2024) reproduction attempts, several of the evaluations they reproduced reported scores higher than the maximum they could obtain across five reproduction runs (Kapoor et al., 2024, Section 6). Sit with that. The published number was above the best of five honest re-runs. A reported score nobody can regenerate from the released materials is a weak thing to bet on, whatever its precision.

Then there is the held-out set, or the absence of one. A held-out set is the slice of tasks you keep back from all tuning, so a score on it reflects whether the agent generalizes rather than how well it was fitted to the cases it saw. Agent benchmarks are often only a few hundred samples, and on many of them, without a held-out test set, a lookup table could in principle score 100 percent (Kapoor et al., 2024, Section 5). A lookup table. That is the ceiling on what a fully visible set can prove. And it is not a hypothetical corner case: in their survey of 17 benchmarks, 7 had no held-out set at all (Kapoor et al., 2024, Section 5, Table 1; count in Table A4). Even a hold-out only protects you if it is genuinely held out from the developers, the tuning, and the pretraining data, and it stops being confirmatory once repeated public submissions turn it into a feedback channel people adapt to (Kapoor et al., 2024, Section 5). The consequence is concrete: on WebArena, the STeP agent reaches 35.8%, but it hardcodes task-specific policies, and on a hold-out of different tasks agents like STeP would score far lower (Kapoor et al., 2024, Section 5). A small, static, fully visible set cannot tell an agent that generalizes apart from one tuned to it. A hidden, refreshed, or shifted set can.

There is one line in the paper I keep coming back to. Accuracy alone cannot identify progress, Kapoor et al. (2024) argue, because it can be “improved by scientifically meaningless methods such as retrying” (Kapoor et al., 2024, Section 2). Now, retries are not cheating. Under a fixed attempt or resource budget, retrying is a legitimate choice. The problem is comparison. If the retry budget is not held fixed and disclosed, a higher score is not evidence of progress, only evidence of more attempts. Same knob, two readings, and only the disclosed one carries information.

What a confirmatory agent evaluation fixes before the number exists

Here is where I have to be careful, because this is the part the title could mislead you about. What follows is not something Kapoor’s paper prescribes. Kapoor documents the failures. The fix I am about to describe, fixing the choices before you can see how they turn out, is ordinary experimental method, and I am asserting it on that basis, not putting the words in Kapoor’s mouth. With that said, here is what a confirmatory, decision-grade evaluation does.

It starts with a hypothesis fixed in advance: a specific claim, stated before the data is collected, in a form that could turn out to be wrong. The strong form is pre-registration, public and time-stamped. You commit the plan (hypothesis, verdict rule, procedure) publicly while keeping the test content sealed, so the public part is the plan, not the hold-out (Nosek et al., 2018). A time-stamped plan does not stop you deviating, and that is not its job. Its job is to make departures visible and labeled, so exploratory work cannot be quietly repackaged as confirmatory after the fact. Attached to it is a locked verdict rule that says what counts as support and what counts against. A threshold picked early is still arbitrary unless it is tied to the quantity, the decision it feeds, and the uncertainty you expect. Concretely: decide in advance that you will call system B better than A only if its estimated advantage clears a margin you set before seeing results. And there is a pre-committed rule for failure: what result would count against the claim. One failed or inconclusive study need not retire a hypothesis for good, because power or measurement validity may still be in question. But what the result means is fixed before you see it, not negotiated after.

The material has to be pinned down too. A version of the task set is frozen during the evaluation, fixed once measurement begins, so nobody can quietly add easy cases or drop hard ones after seeing early results; between rounds the set is refreshed to resist contamination. Part of it is held out, and the confirmatory number is the one measured on a portion genuinely untouched by tuning, not a headline blended over tuned and untouched tasks. Remember the lookup table: the absence of a held-out set is what can in principle let one score 100% (Kapoor et al., 2024, Section 5). Freezing a version during a run is what stops the set itself from turning into one more tuning knob.

Finally, exploratory work and confirmatory work are kept separate, and this is the distinction I would tape to the wall if I could. Exploration, where you try things and generate hypotheses, is indispensable, and I would never tell anyone to stop. But a pattern you found by looking cannot also be the thing you claim to have tested, because you chose it precisely because it looked good. Confirmation is a fresh test, on fresh data, of a hypothesis fixed in advance. Collapse the two and a real-looking result turns out to mean nothing.

None of this describes any particular pipeline. What matters is only that the choices were fixed before the results could influence them, whether through public pre-registration or an independent test on data the builder never touched. And even that is necessary, not sufficient. A study can be fixed to the letter and still be invalid: if the task set does not represent real use, if the tasks miss the capability you claim, if the hold-out is contaminated, the grader unreliable, or too few tasks were run to tell signal from noise. Fixing the choices in advance removes post-hoc tuning. It does not confer validity on its own. Pre-commitment is the floor, not the ceiling.

Why the difference decides whether the number supports the decision

Why does the ordering matter so much, results-first versus commitment-first? Because of a problem psychologists named a while ago: between a question and a number sit many defensible choices (Simmons et al., 2011). Each one is arguable in isolation. But make them while watching the number move and you can walk the result toward the answer you wanted without ever lying, just taking the better-looking option at each fork. That is the deliberate version: undisclosed flexibility a reporter can exploit on purpose.

Two more mechanisms are worth naming, and the first one is what unsettles me most, because it does not require any bad intent. It is the garden of forking paths, where the analysis is chosen in response to the data without anyone consciously laying out the options, so even a single honest-looking path inflates the finding (Gelman and Loken, 2014). You are not cheating. You are just responding to what you see, and that is enough. The second is Goodhart’s law, the familiar observation that a measure tends to degrade once it becomes the target, because effort flows into the measure rather than the thing it stood for. A leaderboard makes the number the target by construction, so all of this pressure lands squarely on it.

There is a second layer, and it compounds the first: a single agent run is noisy. On SWE-Bench-Verified (500 tasks), the per-configuration run-to-run range in single-run pass@1 was 2.2 to 6.0 percentage points across the twelve configurations tested (3 models, 2 scaffolds, 2 temperature settings) (Bjarnason et al., 2026, Table 1). Scope that carefully: it is coding-agent pass@1 noise on one benchmark, not a universal constant. But the shape of it travels. A single run is one draw, and which draw you report can move the number on its own. Repeated measurement answers that noise, though runs only ever give you observations; you still have to aggregate them into an estimate with its uncertainty and judge that estimate against a rule set in advance. And here is the trap inside the trap: repeating runs on a fixed task set reduces run-to-run noise, but it says nothing about whether the chosen tasks represent your deployment, which is a separate question the uncertainty also has to carry. Free choice among noisy runs is how noise gets laundered into a result. We work through the run-to-run part in our analysis of agent evaluation variance.

The steelman: leaderboards have been useful

Let me put the strongest version of the objection, because I think it is largely right and I do not want to win by ignoring it. Shared benchmarks have been genuinely useful. They gave a scattered field a common target, made results comparable across groups, and turned vague improvement claims into something you could actually check, and plenty of real iteration happened on public boards. Told to stop competing and start committing their choices in advance, a lot of good engineers will hear moralizing from someone who has never had to ship. Fair.

So here is where I land: I agree with the premise and reject the conclusion. An adjustable, still-tunable number is a fine instrument of exploration. It surfaces promising directions and exposes weak methods fast, and I would not give that up for anything. The failure is not the format. It is asking that same number to also serve as a confirmatory decision input, which is a different job. Explore with adjustable numbers. Decide on numbers that were fixed before the results could move them.

Adjustable versus fixed-in-advance, side by side

One thing to hold onto before you read the table: the split is about the procedure, not the label on the tin. A rigorously run board belongs on the right, a sloppy experiment on the left. The word “leaderboard” tells you nothing on its own.

DimensionChoices still adjustable after resultsChoices fixed before results are visible
The procedureCan be tuned once the number is in viewCommitted and, in the strong form, time-stamped
Task, prompt, runsThe reporter’s, undisclosedCommitted and disclosed in advance
The task setMay be small or static, sometimes no held-out portionFrozen version, with an untouched held-out portion
What a high number showsThis system scored high, once, this wayA pre-stated claim survived a test that could have failed
What neither guaranteesThat the score generalizes or was not tuned to the setThat the design was valid: representative tasks, valid grader, enough data
Best useExploration: finding promising directionsConfirmation: deciding what to trust

The standard we are building toward

So what can you do with all this tomorrow morning? Before you trust a number, ask one question of it: were the choices behind it fixed before the results appeared, or after? If after, treat it as a signal of where to look, not a verdict. That question costs you nothing and it filters most of the trouble above.

And it is the same bar we are holding ourselves to. We are early, and we have not published a scored benchmark edition yet. When we do, we intend our public numbers to meet this standard, and to report their uncertainty rather than tuck it away. Our methodology page lays out what we publish and how.

Frequently asked questions

Why are AI agent benchmarks misleading?

Because the number can reflect choices the reporter could adjust: which tasks, which prompt, which runs to count. In their reproduction attempts, Kapoor et al. (2024) found several evaluations reported scores higher than anything they could reproduce within five runs, that 7 of 17 surveyed benchmarks had no held-out set, and that accuracy can be “improved by scientifically meaningless methods such as retrying” (Kapoor et al., 2024, Section 2). A high score is exploratory until the procedure behind it was fixed in advance.

What is pre-registration in evaluation?

Pre-registration means committing the hypothesis, the verdict rule, and what result would count against the claim before the results can influence those choices, then measuring against that fixed standard. It removes the freedom to relax the criterion once results are visible. It is a general principle of experimental method, not specific to any one benchmark.

How should I evaluate an AI agent for production?

Separate exploration from confirmation. Use leaderboards to find candidates, then run a confirmatory evaluation with the hypothesis and verdict rule fixed in advance, and the reported figure measured on a held-out portion the builder cannot tune against, with repeated runs aggregated into an estimate and its uncertainty. Ideally an independent evaluator runs the final test, with a validated grader, on a hold-out the builder cannot see. Choose a baseline, and tie the decision rule to what a wrong answer costs. Decision-grade is relative to your claim, not a universal certificate.

Are leaderboards useless?

Not at all, and I would push back on anyone who says so. They are strong tools for exploration: they make results comparable and surface weak methods fast. The error is not the format, it is asking a number whose choices were still adjustable to serve as a confirmatory decision input. Use them to find directions worth testing, and confirm on numbers fixed before the results could move them.

Can a reproducible number still be meaningless?

Yes. Reproducibility means others can regenerate your reported result from your code and data. For a stochastic evaluation, our reading is recovering a compatible estimate, one that agrees within the uncertainty each reports, not an identical decimal. Either way it says nothing about whether the number tests the claim it appears to test. If the tasks, prompt, and counted runs were tuned until the score looked right, you can reproduce a result that was optimized, not tested. Meaning comes from fixing the choices in advance, not repeating the number.

References

  • Kapoor, S., et al. (2024). “AI Agents That Matter.” arXiv:2407.01502. https://arxiv.org/abs/2407.01502
  • Bjarnason, E., et al. (2026). “On Randomness in Agentic Evals.” arXiv:2602.07150. https://arxiv.org/abs/2602.07150
  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632
  • Gelman, A., & Loken, E. (2014). “The Statistical Crisis in Science: Data-Dependent Analysis, a Garden of Forking Paths, Explains Why Many Statistically Significant Comparisons Don’t Hold Up.” American Scientist, 102(6), 460-465. https://doi.org/10.1511/2014.111.460
  • Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). “The preregistration revolution.” PNAS, 115(11), 2600-2606. https://doi.org/10.1073/pnas.1708274114