A pollster-style rating for AI benchmarks

At a glance

What is rated

Benchmarks, not AI models. Each of the 60 benchmarks gets one rating.

The score: reliability

A number from 0 to 1. Take every AI model that has a score on a benchmark. Reliability is the share of the variation in those scores that is explained by the AI models’ general capability. The rest is noise. A reliability of 0.9 means 90% of the variation reflects real differences in capability and 10% is noise.

Grades

A, B, C or D, from the reliability and how certain it is. A benchmark gets a grade only when its reliability is at least that grade’s threshold with 90% probability: A at 0.85, B at 0.70, C at 0.50.

Uncertainty

Every reliability is reported with a 90% credible interval: given the data, there is a 90% probability that the true value lies inside it.

Data

2,898 published scores for 274 AI models on 60 benchmarks, from Epoch AI.³

How good are benchmarks at distinguishing AI model capability?

When two AI models get different scores on a benchmark, part of the difference is real: one model is more capable than the other. Part of it is noise. Noise here means everything other than general capability:

• which questions happen to be in the test, and how many there are;
• differences in prompts, tools and settings between whoever ran each evaluation;
• randomness from one run to the next;
• skills that this benchmark rewards and the other benchmarks do not.

Reliability measures how much of the variation in a benchmark’s scores is the real part. On GPQA diamond, 95% of the variation in scores across its 186 AI models is explained by their general capability, so a gap between two models there is good evidence that one is more capable. On HellaSwag, 56% is, and 44% is noise, so the same gap means much less.

How to read it. The curve is the score the rating model expects from an AI model of a given capability. The shaded band is how far real scores scatter around that curve. On GPQA diamond, the band is narrow and the AI models are spread far along the curve, so their scores differ mainly because their capability differs. On HellaSwag, the band is wide compared with how far the AI models are spread, so two models with the same capability can easily score far apart. Hover a dot to see the AI model and its score.

This is the reliability coefficient of classical test theory², applied to benchmarks instead of exams. It is not about how hard a benchmark is or how high AI models score on it. It is about how far a difference in score can be trusted.

How the ratings are calculated

Two kinds of model appear on this page. AI models are the systems being benchmarked. The rating model is the statistical model we fit to their scores to produce the ratings.

The data

Epoch AI collects published scores for AI models on many benchmarks.³ We use the table Epoch builds its Capabilities Index from:⁴ 2,898 scores for 274 AI models on 60 benchmarks. Most AI models have scores on only a few benchmarks, so most of the table is empty (the field moves fast, and new benchmarks and models arrive constantly, so this is expected). The rating model is built to work with a sparse table.

The rating model

The rating model describes every score with a small number of quantities:

1. Each AI model has one capability score. It is estimated from all of that AI model’s results at once, so a model that does well across many benchmarks gets a high capability.
2. Each benchmark has a difficulty, the capability at which an AI model is expected to score 50%, and a discrimination, how steeply expected scores rise with capability around that point. Together they define an S-shaped curve from capability to expected score: the logistic curve of item response theory, which Epoch’s Capabilities Index also uses.⁵
3. Each benchmark has a noise level: how widely real scores scatter around the curve. Scores are modelled with a Beta distribution centred on the curve, which keeps every score between 0% and 100%.

All of these are estimated together, from all 2,898 scores, with Bayesian sampling (the No-U-Turn Sampler, four chains of 1,000 draws). Each draw is one complete set of capabilities, difficulties, discriminations and noise levels that is consistent with the data. Uncertainty comes from how much the draws disagree.

Reliability

For one benchmark and the AI models scored on it, reliability compares two quantities. The capability variance is how much the expected scores (the points on the curve) vary across those AI models. The noise variance is the average variance of a score around its expected value. Reliability is the capability variance divided by the sum of the two:

reliability = capability variance / (capability variance + noise variance)

We compute it for 1,000 of the 4,000 draws, taken at random. The reported value is the average over draws, and the 90% credible interval runs from the 5th to the 95th percentile of the draws. It is the Bayesian counterpart of a confidence interval: given the data and the rating model, the true reliability lies inside it with 90% probability. Benchmarks scored on few AI models get wide intervals.

Grades

A benchmark gets the highest grade whose threshold its reliability exceeds with at least 90% probability, counted over the draws:

A grade D means the data cannot show the benchmark is reliable, often because it was scored on few AI models. It does not mean the benchmark has been shown to be unreliable.

If every AI model had been scored

Reliability depends on which AI models were scored. A benchmark run only on recent frontier models sees AI models that are close to one another in capability. Their expected scores barely differ, so even modest noise is a large share of the variation, and reliability reads low. This is the same effect that makes an exam look unreliable when everyone taking it is equally strong. For example, DeepSWE was scored on 26 AI models whose capabilities vary only 22% as much as those of all AI models in the data (comparing standard deviations). Its reliability is 0.30 among those models and would be 0.80 across every AI model in the data. We report both. The first says how far to trust the score differences that exist today; the second says how good the benchmark would be at telling apart AI models across the whole range.

The ratings

Each row is one benchmark, grouped by grade. The dot is its reliability among the AI models scored on it. The bar is the 90% credible interval. The ring shows where the reliability would move if every AI model in the data had been scored. Inside a grade, benchmarks are listed alphabetically, not ranked. Hover a row to see its numbers.

Grades are groups, not a ranking. When a ring sits far to the right of its dot, the benchmark was scored on a narrow band of AI models and its low reading comes mostly from that. The 6 benchmarks with too few AI models to grade are shown in grey with their interval only.

Testing the rating model on hidden scores

A reliability is only as good as the rating model it comes from, so we tested the rating model on scores it had not seen. We split the 2,805 scores at random into five parts. In each of five rounds we hid one part, fitted the rating model on the other four, and asked it to predict each hidden score. Every score was hidden exactly once.

For each benchmark, held-out skill compares those predictions with the simplest possible prediction: that benchmark’s average score. It is one minus the ratio of their errors. A skill of 0 means the rating model did no better than the average. A skill of 0.5 means its errors were half as large. Errors are measured with the continuous ranked probability score, which rewards predictions that are both close to the truth and appropriately certain.⁶

How to read it. If the rating model is right about a benchmark, a high reliability should go with accurate predictions of its hidden scores, so the dot sits high and to the right. A benchmark the rating model calls noisy should be one whose hidden scores it predicts poorly, low and to the left. A dot far from the dashed trend would be a benchmark the rating model gets wrong. Across 60 benchmarks, the order by reliability and the order by held-out skill have a rank correlation of 0.95, where 1 would mean the same order.

This is a consistency check, not proof. Both numbers come from the same rating model, so an error it makes the same way on every benchmark would not show up here. The comparison in the next section is a stronger test. Separately, a rating model fitted only to AI models released before 2025-09-29 orders the benchmarks much like the full fit (rank correlation 0.79).

Comparison with Epoch’s Capabilities Index

Epoch’s Capabilities Index⁴ places AI models on one capability scale using the same S-shaped curve.⁵ Our version of its form fits the curve by least squares and keeps a single best estimate of every quantity; Epoch’s published index adds bootstrap intervals, which we did not reproduce. The rating model differs in two ways: it estimates a separate noise level for every benchmark, and it carries the uncertainty in every quantity through to its predictions. We refitted Epoch’s form on the same data and gave all four methods the same hidden-score test:

• The rating model: the Bayesian rating model described above.
• Epoch Capabilities Index form: the same curve fitted by least squares, with a fixed error band per benchmark.
• Average standardised score: each AI model’s average standardised score across its benchmarks, mapped to each benchmark with a straight line.
• Guess the benchmark’s average: always predicting the benchmark’s average score.

The left panel is the typical prediction error in percentage points; shorter is better. The right panel is how often each method’s 90% prediction interval contained the true score. An honest 90% interval should contain it about 90% of the time.

The two curve-based methods are almost equally accurate: the rating model’s error is 0.4 points lower (95% interval 0.3 to 0.5). The difference is in the intervals. The rating model’s 90% intervals contained the true score 89% of the time. The least-squares fit’s contained it 84% of the time, so it is more confident than the data justify. The gap widens on the newest AI models: when half the scores of the most recent quarter of AI models are hidden, the errors are 5.4 and 6.1 points and the coverage is 91% and 82%. Reliability is a statement about noise, so it needs a model whose uncertainty is right. That is what the rating model adds; it is not a better capability index.

Limitation: capability is not exactly one number

The rating model gives every AI model a single capability score. If that assumption were exactly right, the part of each score the model cannot explain (the score minus its expected value) would be pure noise, and those leftovers would be unrelated from one benchmark to the next. If AI models also differ in some second skill, say coding or knowledge recall, the leftovers on two benchmarks that both reward that skill would rise and fall together.

We tested this directly. For each of the 509 pairs of benchmarks that share at least 12 AI models, we computed the correlation between their leftovers and averaged its square over all pairs. In the real data the average is 0.089. We then simulated 500 tables from the fitted rating model itself, where one capability explains everything by construction, and computed the same number: 0.046 on average, and only 0.2% of simulated tables reached the real value. So the real leftovers move together about 1.9 times as much as one capability allows: there is real structure beyond a single number.

The pairs whose leftovers move together most are LAMBADA and PIQA; HellaSwag and TriviaQA; HellaSwag and PIQA; ProofBench and Remote Labor Index. Most are older language-understanding benchmarks scored on the same group of older AI models. One candidate explanation is guessing: on a multiple-choice test, a model with no relevant skill still scores above zero. Refitting with a separate chance-level floor for every benchmark leaves the excess almost unchanged (0.088 against 0.046), so guessing is not the cause.

What this means for the ratings: if a benchmark mainly measures a skill that the other benchmarks do not, the rating model counts that skill as noise and the benchmark’s reliability reads low. In our simulation study, adding a strong second skill had exactly this effect, and the held-out tests above did not catch it. A grade therefore says how well a benchmark measures general capability, the factor all 60 benchmarks share. It is not a verdict on whether the benchmark is useful.

Data and reproducibility

Data: Epoch AI, ‘Capabilities & benchmarking’, CC BY 4.0.³ We use the table Epoch builds its Capabilities Index from,⁴ so every benchmark here is one Epoch already considers suitable for that index: 274 AI models, 60 benchmarks and 2,898 scores. The sampler converged: the worst R-hat across every benchmark’s parameters is 1.004, below the 1.01 limit for grading.⁷ Data fingerprint: d5782d112cc0fa89.

The method and the tests it had to pass were written down before the first fit, and every later change is recorded as a dated amendment. The repository has the code, the exact Epoch table we used, every number on this page, the full method, a simulation study that checks the method on data where the true reliabilities are known, and the complete list of limitations.

References