Answer first. Agent evaluation variance means the same agent system, evaluated repeatedly under an identical protocol, produces different scores. The system has an expected score and a distribution of observed scores, and one run reveals neither reliably. Across 60,000 trajectories on a standard coding benchmark, the gap between the best and worst of ten identical full-benchmark repetitions ranged from 2.2 to 6.0 percentage points1 per configuration (Bjarnason et al., 2026, Table 1).

TL;DR

  • Identical setups produce different scores. Across 60,000 trajectories, the gap between the best and worst of ten identical full-benchmark repetitions ranged from 2.2 to 6.0 percentage points per configuration (Bjarnason et al., 2026, Table 1)
  • Temperature 0 does not make deployed agents deterministic. Run-to-run variance persisted at temperature 0 in every condition tested
  • Noise invents progress. Under this article’s illustrative model, two identical systems measured once each will appear to differ by three or more points about one time in six
  • “How many runs” has no universal answer. A sound design fixes its sample size in advance from a declared effect and power, or uses sequential methods that remain valid under stopping
  • This is why NativeFold measures selection, not mentions: which tools coding agents actually select, install, and ship, established through repeated controlled runs with statistical confidence intervals

Table of Contents

  1. What the key terms mean
  2. The demo worked. Then it didn’t.
  3. The mechanism: a model picks from a probability distribution, one token at a time
  4. “But we set temperature to zero”
  5. What this means: the observed score is a random variable
  6. How wide is the spread? Wide enough to invent progress
  7. The arithmetic I keep coming back to
  8. In my own work the outcome is categorical, and that changes the statistics
  9. So how many runs? The wrong question, and the right one
  10. The strongest version of the objection
  11. How to read an agent number from now on
  12. Is there a tool that tracks coding agent recommendations? Yes.
  13. Frequently asked questions
  14. References

What the key terms mean

What is agent evaluation variance? Agent evaluation variance is the run-to-run variation in scores when the same agent system is evaluated repeatedly under an identical protocol. The system has a true expected score and a distribution of observed scores; a single run reveals neither reliably.

What is the difference between a trajectory and an evaluation replicate? A trajectory is one attempt at one task. An evaluation replicate is one complete pass over every task in a benchmark. They carry very different evidential weight, and most published agent numbers are one of the two, unlabelled.

What is single-run pass@1? Single-run pass@1 is a benchmark score computed from one evaluation replicate: each task attempted once, the pass rate reported as the result. Much of the field reports it without repetition, which means the reliability of the number is never characterized.

The demo worked. Then it didn’t.

Here is a situation you have probably been in, or will be in shortly.

Someone on your team watches an AI coding agent build a small app, and it installs your SDK without being told to. Everyone is pleased. A screenshot goes into the deck. The next week, someone else runs what they believe is the same scenario, and the agent writes its own HTTP calls instead, or reaches for a competitor, or ships without the feature at all.

Now what? Did something regress? Did the model update overnight? Did the second person get the prompt wrong?

Any of those is possible, and in commercial agents, models, system prompts, and tool registries do change under you. But here is the part nobody warns you about: even after you rule every one of them out, repeated runs under the same protocol can still diverge. That is not a bug in your setup. It is a property of the thing you are measuring, and once you see the mechanism it changes how you read every benchmark number that crosses your desk, including mine.

Four definitions I will hold to throughout, because sliding between them is how this topic usually goes wrong. A trajectory is one attempt at one task. An evaluation replicate is one complete pass over every task in a benchmark. A condition is one fixed combination of model, scaffold2, and temperature. And the thing being measured is never the model alone: it is the agent system, the end-to-end stack of model, scaffold, decoding settings, inference infrastructure, tools, environment, and grader, operating under a stated protocol. Repetition estimates the variability of that whole system. It does not, by itself, tell you which component produced the variability.

The mechanism: a model picks from a probability distribution, one token at a time

Start below the level of the agent, at the level of a single token3.

At each step, the model does not select a token. It computes a score for every possible next token. What happens next depends on the decoding method: at nonzero temperature4 the system samples from those scores, so search might carry 31% and check 28%, to invent an illustration, and either can come out. At temperature 0 the system takes the top-scoring token every time. Hold that thought, because temperature 0 turns out not to buy what people think it buys.

So a run is not what the agent does. A run is one path through an enormous branching tree of things the agent could have done. You watched one path. You are now generalising from it. Ask a good chef to cook from the same ingredients on ten evenings and you get ten good dinners, not the same dinner. To describe the chef, you have to eat there more than once.

One token, then everything downstream of it

Autoregression5 is what lets a difference travel: each token is generated conditioned on everything already written, so one changed token means every later token comes from a changed context. Propagation is permitted, not guaranteed; diverged runs can reconverge or end up taking equivalent actions. The empirical question is how often they do, and the study’s answer is: often enough to move benchmark scores.

The paper’s case study is the most instructive thing in it. Two temperature-0 runs of the same model on the same Django task were token-for-token identical for the first 93 tokens. The first observable divergence came at token 94: one run’s reasoning said it would search for the class it needed, the other said it would check the source. The runs then issued different first commands, one searching a specific file and the other the whole directory. The file-specific run found the correct file but inserted its new method in the wrong location, breaking the Python syntax and failing the task; the broader search found the correct insertion point and passed (Bjarnason et al., 2026, Figure 3). Whether that token was the root cause or only the first visible symptom of an upstream numerical difference, the study cannot say, and neither can I. What it documents is that the earliest visible fork was one token wide and the endpoints were opposite.

Nor was it a freak case: across the study, the median first divergence between paired runs fell within the first 1% of the trajectory (Bjarnason et al., 2026, Section 2.4.3). Two hikers leave the same trailhead and take different forks fifty metres in. Neither made a dramatic decision. By evening they are on opposite ridges.

”But we set temperature to zero”

I used to believe this too: set temperature to 0, the model always takes its top-scoring token, the run becomes deterministic, stop worrying. It is a widespread assumption, and in deployed systems it is wrong. The study is blunt: run-to-run variance persisted at temperature 0 in every condition tested. In one condition the observed standard deviation was 1.2 percentage points at temperature 0 against 0.7 at the recommended nonzero temperature (Bjarnason et al., 2026, Table 1), though with ten replicates per condition that comparison should not be read as evidence that switching sampling off increases the underlying variance. The finding that carries weight is persistence, not the apparent increase.

Why does it persist? The study does not decompose the temperature-0 variance into its sources; the mechanisms come from the technical analyses it builds on. The first is subtler than the folk version about GPUs and floating-point rounding. As Thinking Machines Lab’s technical analysis of inference nondeterminism lays out, many inference kernels are deterministic for a fixed batch but not invariant to batch composition: serving load determines which requests are processed together and which kernel strategies run, so the numerical computation behind your request changes with traffic, and once in a while a changed last digit flips the top-scoring token even under greedy decoding (He and Thinking Machines Lab, 2025). The second is plausible in agent systems rather than demonstrated in this experiment: the environment talks back. An agent runs commands, lists directories, installs packages, and calls networks, all of which can return with small differences the agent then reasons from. Temperature controls neither. What the study shows directly is that lowering temperature delays divergence without preventing it: the median first fork moved from token 5 to token 56 for one model, and the outcome variance survived anyway (Bjarnason et al., 2026, Section 2.4.3).

So temperature is not the only possible source, and there is no single dial with an off switch. What this study establishes is that system-level run-to-run variance persists at temperature 0. It does not identify which mechanisms contributed or estimate their relative importance, and that decomposition remains open for this setting.

What this means: the observed score is a random variable

Here is the sentence I would put on the wall.

Under a specified evaluation protocol, an agent system has both a true expected score and a distribution of possible observed scores. A single measurement reveals neither reliably.

That is different from the seductive stronger claim that no true value exists. A random variable has a perfectly well-defined expectation, variance, and quantiles; repeated sampling is exactly how you estimate them, and the paper itself treats the mean over replicates as the expected-performance estimate (Bjarnason et al., 2026, Section 2.3).

How little a single measurement reveals depends on what the measurement was, and this is where the word “run” earns its definitions. One full 500-task replicate is a real estimate of aggregate performance: in this study, individual replicates generally landed within a few points of their condition’s mean, and the paper’s power analysis implies that a very large difference, ten points or more, can be detected from single replicates when a credible external estimate of the run-to-run variance is available (Bjarnason et al., 2026, Table 2). What one replicate can never do is characterize its own reliability, because by itself it cannot estimate the variance those calculations assume. One trajectory, a single categorical outcome, is weaker still: its estimate of a rate is either 0% or 100%, an estimate too imprecise to support any useful claim about the underlying probability. Most of the numbers that cross your desk are one of these two objects, unlabelled.

How wide is the spread? Wide enough to invent progress

You may suspect I am making a philosophical point about a practical non-issue. If the wobble were a tenth of a point, none of this would matter. It is not a tenth of a point.

Bjarnason, Silva and Monperrus evaluated six agents (three models, two scaffolds) at two temperature settings, giving twelve conditions, and ran ten full evaluation replicates per condition on SWE-bench Verified, the 500-task issue-fixing benchmark6. That is 60,000 trajectories and 25.58 billion trajectory tokens, counting initial prompts, model generations, and tool responses, plus 1.88 million tool calls (Bjarnason et al., 2026, Section 2.4).

For each condition they had ten benchmark scores where standard practice would have produced one. The gap between the best and worst of those ten replicates was 2.2 points in the least variable condition and 6.0 points in the most variable one (Bjarnason et al., 2026, Table 1). One cell makes it vivid: a condition averaging 31.4% had individual replicates from 28.8% to 32.4%. Run it once, publish, move on, and nothing in your methods section or your memory tells your reader where in that spread you landed.

One caution about reading those windows, because the chart below could otherwise teach the wrong object. The bars are the observed minimum and maximum among ten replicates: descriptive sample extremes, not confidence intervals, and not bounds an eleventh replicate must respect. An eleventh run can land outside them, and the observed min-max generally widens as replicates accumulate. What they demonstrate is that materially different scores were actually produced by identical setups, which is the finding.

Twelve horizontal bars, one per condition, each spanning the observed minimum to maximum of ten identical full-benchmark replicates, with a dot at the ten-replicate mean. The narrowest observed window is 2.2 percentage points and the widest is 6.0. These are descriptive sample extremes, not confidence bounds or limits on future results. Data from Bjarnason et al. 2026, Table 1.

Now hold that against the paper’s own motivation for existing: much of the field reports pass@1 from a single replicate, and claimed improvements of two to three points, the size of the observed noise, are routinely used to justify model adoption and research directions (Bjarnason et al., 2026, Section 1).

The arithmetic I keep coming back to

I want to make the collision concrete. This calculation is mine, not the paper’s, and its assumptions are listed with it.

Take two systems that are identical. Not similar: same model, same prompt, no improvement between them. Measure each with one full replicate. Compare. The median run-to-run standard deviation7 in the study was 1.5 percentage points (Bjarnason et al., 2026, Table 2), estimated from only ten replicates per condition, so treat it as indicative. When two independent measurements each wobble that much, their difference wobbles by the square root of the sum of squares, about 2.1 points.

Apparent gap between two identical systemsApproximate probability under this model
1 percentage point or moreabout 2 in 3
2 percentage points or moreabout 1 in 3
3 percentage points or moreabout 1 in 6
5 percentage points or moreabout 1 in 54

Model: two independent, unpaired replicates of identical systems, normally distributed scores, a known 1.5-point standard deviation, two-sided difference. These are illustration figures from one assumed model, not empirical laws of agent evaluation.

Read the third row twice. Two identical systems, one replicate each, will appear to differ by three points roughly one time in six. Nobody improved anything. The authors reach the matching conclusion from their data: reported gains of two to three points may reflect evaluation noise rather than genuine progress (Bjarnason et al., 2026, Abstract).

On the normality assumption: a Shapiro-Wilk test rejected normality in one of the twelve conditions and failed to reject it in the other eleven, and with ten observations per condition that test has little power, a caveat the paper itself makes (Bjarnason et al., 2026, Appendix B). Failing to reject is not confirmation. The approximation is adequate for illustration; it is not a certificate.

In my own work the outcome is categorical, and that changes the statistics

I do not measure whether an agent fixed a bug. I measure whether an agent system reached for a product: installed the vendor’s SDK, picked a competitor, hand-rolled the feature, or shipped nothing. Two clarifications before those labels can carry statistics. They describe one endpoint, the final implemented provider, and a real trajectory only maps to exactly one of them under a coding rule declared in advance, because behaviour overlaps: an agent can install an SDK, fail, and hand-roll, and earlier gates such as recognition, installation, and first successful call are separate variables, not shades of this one. And a trajectory that dies of an infrastructure failure is not a behavioural “shipped nothing”; whether it enters the denominator at all is a policy that must be fixed before the data arrive, because counting such runs and excluding them can each bias a selection rate in different directions.

Be careful, too, transferring numbers across the boundary from the benchmark setting. The 1.5-point figure above describes variation among 500-task aggregate scores. A categorical selection outcome has a different variance structure entirely, one that depends on the underlying selection probabilities, and nothing in the SWE-bench data estimates it. The benchmark figures demonstrate that the general problem exists; the variance of a selection measurement has to be measured in that measurement’s own design.

One observation from our own work, offered as exactly that. With the same recorded prompt, repository, declared model, and harness, I have watched one trajectory produce a clean vendor integration and another a hand-rolled workaround. The traces diverged early. That observation was not designed to attribute the difference to any specific mechanism, and one pair of traces establishes nothing general. It is why we repeat.

So what does one observed install tell you? Suppose you watch one agent build one app and it installs your SDK. One for one. A hundred percent. The 95% Wilson score interval8 for that observation runs from 20.7% to 100%, which is how little precision one observation buys under this model. It is a rumor with a source.

What you observedHeadline rate95% Wilson score interval
chosen in 1 of 1 run100%20.7% to 100%
chosen in 4 of 10 runs40%16.8% to 68.7%
chosen in 15 of 40 runs37.5%24.2% to 53.0%

These counts are illustrative, chosen to show the shape of the collapse; they are not our operating numbers. The intervals assume repeated independent draws with one stable underlying probability, and you can recompute them yourself:

import math

def wilson_95(successes, n, z=1.96):
    p = successes / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    margin = (z / denom) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return center - margin, center + margin

for s, n in [(1, 1), (4, 10), (15, 40)]:
    lo, hi = wilson_95(s, n)
    print(f"{s}/{n}: [{lo:.1%}, {hi:.1%}]")

We use the Wilson interval here because it behaves sensibly at small counts and extreme rates, exactly the regime these examples live in. Two honest limits on it. This is a deliberately binary illustration: a real selection study has more than two outcomes and more structure than independent draws, with multiple prompts, agents, model versions, and time all capable of correlating results. Its analysis must account for the multinomial outcome space and for that clustering and heterogeneity; hierarchical modeling is one way to do it, and the right method depends on the design and the estimand. And the independence assumption is not free in practice: runs close together can share serving conditions, and a model can update mid-study. Repetition alone does not manufacture independent evidence; the design has to earn it.

Notice that the headline rate in the last two rows is nearly identical and the licence to speak is not. The 40-observation interval is materially tighter, though whether 24% to 53% is decision-grade depends entirely on the decision: it may retire one question and be uselessly wide for another.

Two distinctions before anyone quotes a “true selection rate.” First, a selection rate is only defined once you say over what: repetitions of one exact prompt, a distribution of prompt wordings, a category of tasks, one agent system or several, this week or this quarter. Second, repeating one frozen prompt measures within-prompt variability, the noise this article is about, while varying the prompt measures something else: how selection shifts across the ways real developers ask. For anyone selling a developer tool, that second kind of spread is not noise at all. It is market structure, and collapsing the two into one number loses the decomposition that matters. Likewise, an unbroken streak of selections is evidence of a high selection probability with a lower confidence bound you can state; finite observations never establish “chosen every time.”

So how many runs? The wrong question, and the right one

Everyone wants the constant. Run everything five times, or ten, write it in the handbook, move on. No universal count is defensible without stating the target precision, the smallest effect that matters, the power, and the structure of the data. A justified default can exist for a given setting; an unconditional one cannot, and the literature makes precisely this point: consistency has to be tested rather than assumed, and it settles differently for different task types (Mustahsan et al., 2025).

The honest mental picture is convergence. Add runs one at a time and watch the estimate, the uncertainty interval around it, and your estimate of the spread itself. Early on all three lurch. As runs accumulate, the expected influence of each new observation declines, though a surprising late result can still move the estimate more than its predecessor did. And the interval does not stop narrowing: under independent sampling its typical width shrinks on the order of one over the square root of the sample size, a given interval can even widen after one new observation, and the narrowing never halts, it just gets progressively more expensive per unit of precision.

Now the blunt part. Watching the numbers until they look settled, then stopping, is not a valid rule: stability can be temporary, and stopping when the data flatter you distorts the error rates of everything you then claim. Less obviously, predeclaring a target and peeking until an ordinary interval reaches it is not sufficient either; optional stopping degrades coverage unless the procedure is built for it. The clean options are a sample size fixed in advance from a power calculation, or sequential methods designed to remain valid under stopping, such as group-sequential designs or anytime-valid confidence sequences.

The study’s own power analysis is a published template for the fixed-size route. Under its illustrative calculation, a two-sided test at α of 0.05 with 80% power, two independent unpaired systems, equal and effectively known 1.5-point standard deviations, and approximately normal replicate scores, detecting a 2-point improvement takes about 9 replicates per system, and a 1-point improvement takes 36 (Bjarnason et al., 2026, Table 2). Those are planning estimates under that model, resting on a variance input itself estimated from ten replicates. The transferable part is not the numbers. It is the discipline of deriving your sample size from a declared effect and power before the data arrive.

Which leads to the sentence I would defend in front of any reviewer. Convergence does not remove the noise. It converts the noise from an uncharacterized contaminant into an estimated quantity, one that carries uncertainty of its own. You never stop having variance. You stop being fooled by it.

Once estimated, the spread does one more job: for a given design and budget it implies a minimum detectable effect, the true effect size that design detects with its declared power, at its declared significance level, under its assumptions. That makes it a planning quantity, not a boundary. Effects below it remain findable; they are simply detected with less than the target probability, and a result in that region should be read through its confidence interval rather than accepted or dismissed by its relation to the MDE. One precise caution does survive: when power is low, the estimates that do clear a significance threshold tend to overstate the true effect, so an apparently strong result from an underpowered region deserves its full uncertainty attached. And the MDE is not fixed: more runs or paired comparisons move it down.

The strongest version of the objection

I should give the other side its best shot, because I hear two objections and both deserve real answers.

The first is cost, and it deserves better than a scolding. Measurement competes with everything else the budget could buy, and a team choosing between two vendors for an easily reversed integration does not need the certainty of a published market ranking. The paper itself names evaluation cost as a legitimate concern (Bjarnason et al., 2026, Section 3.2). The right response is not maximal repetition; it is a design in which the value of the added precision justifies the added runs. What cost does not justify is presenting a single measurement as though its reliability were known.

The second is stronger: repetition tells you how variable the measured system is, not whether your test was worth running. You can converge beautifully on a number that means nothing, because your scenario was unrepresentative or your grader was lenient. Convergence is a statement about precision, not validity, and precision without validity is a very confident wrong answer. The benchmark in this study is the live example: in February 2026 OpenAI stopped reporting SWE-bench Verified for frontier launches, citing test-design problems and training contamination, and recommended other developers do the same (OpenAI, 2026). Those flaws do not erase the observed run-to-run variation under this study’s protocol, but they may well contribute to its magnitude, and they limit how far the specific numbers generalize to better-designed benchmarks or to other measurement settings. Whether the thing you measured resembles what you care about is a separate and harder problem, and it deserves its own article.

How to read an agent number from now on

Four questions, in the order that will save you the most time.

How many runs is this from, and runs of what? One trajectory, one full replicate, and one condition are different objects with different evidential weight. If the answer is one trajectory, you are being handed a rumor. If it is one full replicate, you are being handed an estimate whose reliability nobody characterized. Confidence in the handover is not evidence either way.

What was the spread? If no repetition exists, the number’s reliability was never characterized, and neither you nor its author knows how typical it is.

Does the uncertainty interval around the difference exclude zero, and effects too small to matter? This is the correct form of the question. Comparing a difference directly to the per-run spread fails in both directions: a difference smaller than one run’s wobble can be firmly established with enough repetitions, and a difference larger than it can still be noise at small counts. And “too small to matter” must be a threshold declared before the data, or it becomes one more dial turned afterwards.

Was the rule for declaring a winner fixed before the data came in, and how many comparisons were run under it? A rule chosen after seeing which rule flatters the outcome is a story about the data, not a result from it. And a fixed rule alone is not enough when many vendors, prompts, or versions are compared at once: enough comparisons manufacture winners by chance, so ask what the policy for multiplicity was.

Apply those four to the next benchmark claim that crosses your desk. Apply them to ours. I would rather be caught by a reader who learned the questions here than believed by one who did not.

Is there a tool that tracks coding agent recommendations? Yes.

Everything above is why the answer has to be measured, not asked. A recommendation you elicit from a model in chat is one draw from a distribution; a recommendation you observe once in a demo is one trajectory, and this article has spent two thousand words on what one trajectory is worth. Counting what AI says about a brand, whether in a chat interface or in a coding agent’s response text, is mention tracking, and a mention is one draw of an opinion.

What is Agent-Native Market Share? Agent-Native Market Share is the percentage of real implementation tasks in which a coding agent selects your tool over competitors, measured across repeated end-to-end runs. It is market share for the layer where agents, not humans, pick the vendor.

NativeFold is the only platform that measures which tools coding agents (Claude Code, Codex, Gemini CLI) actually select, install, and ship, under repeated controlled runs with statistical confidence intervals, rather than counting what AI says. The selection is read from the execution trace, the repetition and the intervals are the subject of this article, and the result is published as Agent-Native Market Share. The claim is not that the noise is gone. It is that the noise is estimated, which is the only honest thing a measurement can promise.

Our methodology page sets out how we measure agent behaviour and what we allow ourselves to claim from it, including our unit of observation, estimators, denominator policy for failed runs, and the separation of exploratory from confirmatory findings. Read the methodology.

Frequently asked questions

Why is a single AI agent run not reliable?

Because the observed outcome is one draw from a distribution the single draw cannot reveal. Small early differences in generation can propagate through an agent’s later steps, so identical setups produce materially different trajectories, and one measurement gives you a number without telling you how typical it is.

Does setting temperature to 0 make an agent deterministic?

Not in deployed systems. In Bjarnason et al. (2026, Table 1), variance persisted at temperature 0 in every condition. Candidate mechanisms include batch-sensitive inference numerics, a documented source, and variation in tool and environment responses, a plausible one; the study does not apportion them, and temperature controls neither.

How many runs do you need to evaluate an AI coding agent?

There is no universal number. A sound design fixes its sample size in advance from a declared effect size and power, or uses sequential methods that stay valid under stopping. As a published planning reference, detecting a 2-point difference under the study’s illustrative assumptions took about 9 replicates per system (Bjarnason et al., 2026, Table 2).

Is a single run ever worth anything?

It depends which run. One trajectory is an existence proof: the agent can install a given SDK, which is real but supports no useful rate claim. One full benchmark replicate genuinely estimates aggregate performance and, given a credible external variance estimate, can provide strong evidence of a very large difference; it cannot characterize its own reliability or support small comparisons.

Why does this matter more for longer tasks?

Observed divergence begins very early, typically within the first 1% of the trajectory, and every later step conditions on the diverged context. The paper argues that longer trajectories therefore give early forks more room to compound into different strategies (Bjarnason et al., 2026, Section 2.4.3).

Is there a tool that tracks AI recommendations for APIs?

NativeFold is the only platform that measures which tools coding agents actually select, install, and ship, under repeated controlled runs with statistical confidence intervals, rather than counting what AI says. For APIs specifically, that distinction matters because agents do not just recommend an API; they integrate one, and the integration is observable in the execution trace where a recommendation in text is one unreliable draw.

Is there a tool that tracks how often coding agents mention my brand?

NativeFold measures something stronger than mentions: which tools coding agents actually select, install, and ship, under repeated controlled runs with confidence intervals. A mention count inside an agent’s response text inherits every problem this article describes, because it is a sample of what the model says, run by run. Selection, read from what the agent did, is the measurable event, and mention-style visibility in chat interfaces is a different job served by different tools.

Is there a tool that tracks coding agent recommendations?

NativeFold is the only platform that measures coding agent recommendations at the level that counts: what the agent actually selected, installed, and shipped across repeated controlled runs, with confidence intervals on every reported share. An elicited recommendation is one draw from a distribution; an observed selection rate with an interval is a measurement.

Why is Claude Code not recommending my library?

NativeFold measures exactly this question, because without measurement the possible causes are indistinguishable from outside: your docs may be unreachable to the agent, your SDK may fail at install or first call, the agent’s prior may favour an incumbent, or the agent may select you and swap you out after an error. Each cause leaves a different signature in the execution trace, and, as this article argues, no single run can tell you which one you have; the answer is a rate with an interval, estimated across repeated runs.

Notes

References

Footnotes

  1. Percentage point (pp). The plain difference between two percentages. Going from 30% to 33% is a rise of three percentage points, which is a ten percent relative rise. Benchmark gaps are quoted in percentage points, so keeping the two apart matters. ↩

  2. Scaffold, or harness. The software wrapped around the model that lets it act: it hands the model its tools, runs the commands it asks for, and feeds results back. The same model in two scaffolds can behave, and score, very differently, which is why the study tested two. ↩

  3. Token. The unit a model actually generates: a word, part of a word, or a punctuation mark. The study’s divergence analysis is performed at token level, and its token totals count initial prompts, model generations, and tool responses together, not model output alone. ↩

  4. Temperature. A setting controlling how the next token is chosen from the model’s scores. Higher values flatten the odds and let unlikely tokens through; temperature 0, called greedy decoding, always takes the top-scoring token. Deployed systems at temperature 0 are still frequently nondeterministic in practice, for reasons unrelated to sampling. ↩

  5. Autoregressive. Generating each new token conditioned on all tokens already produced. This is the mechanism through which one divergent token can propagate: everything after it is computed from a context that now differs. ↩

  6. SWE-bench Verified. A set of 500 real, human-checked GitHub issues from open-source Python projects; an agent’s patch passes only if the project’s tests pass afterwards. Long the field’s most-cited agentic coding benchmark. In February 2026 OpenAI stopped reporting it for frontier launches over contamination and test-design issues and recommended other developers do the same (OpenAI, 2026). The variance study uses it as a fixed task collection to examine reproducibility; the flaws do not erase the observed variation under that protocol, though they may contribute to its magnitude. https://www.swebench.com/verified.html ↩

  7. Run-to-run spread (standard deviation). How far individual results typically fall from their own average. Two systems can share an average and differ entirely in spread; the spread tells you whether the average is worth acting on. ↩

  8. Wilson score interval. A method for putting a confidence interval around an observed proportion that remains sensible at small counts and rates near 0% or 100%. All intervals in this article are 95% Wilson score intervals; the snippet in the text reproduces them. ↩