None of ARR’s 69,781 Published Paper Scores Is a 5.0
No ACL Rolling Review paper has ever scored a 5.0. Across five years and 69,781 scored submissions, the top of the scale is empty. A 5.0 here is an average, and reaching it would take a paper whose whole review panel converged on a perfect score, which in five years never happened.
On a scale that reports each reviewer separately, top scores do happen. ICLR publishes every reviewer's individual number instead of averaging them, and in 2025 the relighting model IC-Light, by ControlNet's creator, drew a straight 10, 10, 10, 10 from all four reviewers. ARR's averaging into a single 1-to-5 number makes the same outcome almost arithmetically invisible.
I found the empty ceiling while checking whether my own middling scores were harsh or normal. The important detail was how ARR constructs the published number: it averages a paper’s reviews before adding that paper to the histogram.
The data is public. ACL Rolling Review runs a stats dashboard backed by the open acl-org/arr-health repository, which publishes a per-cycle score histogram at iterations/<year>/<month>/review_scores.csv. Every number here is reproducible with a loop over those files.
These are paper scores, not reviewer scores
The number in the ARR histogram is one aggregate score per paper, not one per reviewer. This is the thing I nearly got wrong, and it changes what every chart below means. The histogram's total for each cycle equals the count of active submissions, not the count of reviews, and the real number of individual reviews is three to six times larger.
| Cycle | histogram total | active submissions (papers) | actual individual reviews |
|---|---|---|---|
| 2026 May | 13,668 | 13,668 | 42,250 |
| 2025 Feb | 7,321 | 7,321 | 23,126 |
| 2024 Jun | 4,774 | 4,774 | 15,234 |
The match to the paper count is exact, every cycle. So each row is one number per submission, the aggregate of that paper's three-or-so reviews (which is why the values still land on clean half-point steps). The 69,781 figure is submissions summed across 35 cycles, not reviews, and a resubmitted paper counts once per cycle. The individual reviewer scores that go into each average are not published anywhere.
That distinction matters because it kills the tempting headline. "No reviewer gives a five" is unsupported: individual 5.0s almost certainly exist and get averaged away. What the data actually shows is that no paper ever earns a 5.0 average, which is a claim about how often a whole panel agrees.
No published paper-level average reaches 5.0
Across five years, the top of the aggregate scale is unused. Of 69,781 scored submissions, exactly zero reached a 5.0, and only 28 (0.04%) reached a 4.5. Anything at or above 4.0 is 1.4% of all papers. The nominal 1-to-5 scale behaves as a 2-to-3 scale: 83% of every scored submission lands at 2.0, 2.5, or 3.0, and the single most common score is 2.5.
Part of this is real and part is arithmetic. Averaging three scores mechanically pulls a paper toward the middle: to average a 5.0, every reviewer has to give roughly a 5.0, so the extremes wash out before they reach the histogram. The empty ceiling is therefore weak evidence about how any individual reviewer behaves and strong evidence about how rarely three reviewers agree a paper is flawless. In five years, no paper had every reviewer give it roughly a 5.
The pattern is not an artifact of tiny early cycles. It holds where it carries weight: the November 2021 cohort (2,585 papers), January 2026 (9,177), and May 2026 (13,668) all have zero 5.0s.
Single meta scores have wider tails than averaged review scores
Each paper carries two numbers: the aggregate of its reviews, and a single meta score from the area chair. The meta score reaches the top of the scale far more often. A paper's averaged reviews land at 4.0 or above 0.8% of the time; its meta score does so 7.5% of the time.
Before reading this as "area chairs are the generous ones," note the confound I could not remove. The review number is an average of three scores; the meta number is one person's single score. A single score has more variance than an average of three by construction, so the meta distribution has fatter tails on both ends: it reaches 4.0 more often, and it also drops to 2.0 or below more often (26% versus 23%). Some unknown share of the gap comes from that arithmetic.
What survives the confound is the middle. Averaged reviews pile onto the 2.5 fence: 36% of papers sit exactly there. Meta scores avoid it, dropping the 2.5 share to 22% and pushing that mass outward. Area chairs write the meta score after reading the reviews, so the two numbers are not independent opinions: what this shows is that area chairs, given a set of reviews averaging 2.5, usually pick a side. I restricted this comparison to the half-point-scale era, February 2025 through May 2026 (43,458 papers), because ARR used an integer-only meta scale before then and mixing the two manufactures a difference that is really a scale change.
Mean paper scores stayed between 2.24 and 2.80 as volume grew 100-fold
The one finding here that needs no caveat is stability. A single cycle grew from 23 scored papers in mid-2021 to 13,668 in May 2026, more than a hundredfold, and the mean score did not follow. It stayed inside a narrow 2.24-to-2.80 band the entire time. The two lowest points, October 2023 and July 2025, are small off-cadence cycles, not a trend.
This is the observation David Jurgens flagged from the same public data. I read the flatness less as reassurance than as a property of a compressed instrument. When paper-level scores are averages clustered on a two-point band around "revisions needed," the mean has almost nowhere to move regardless of what gets submitted. A flat average under a 100x load increase is what a compressed scale would produce whether or not quality held.
What this data cannot tell you
These are aggregated per-paper histograms, and the aggregation is the whole caveat. There are no individual reviewer scores, no paper identifiers, no accept or reject labels, no review text. You cannot recover how any single reviewer scores, measure disagreement inside a paper's review set, or join a score to an outcome.
It also cannot answer whether an automated reviewer would reproduce this distribution. There are no matched papers to score, and ACL reviews are public on OpenReview, so a model may have seen a paper's reception during training. A model scoring such a paper could be recalling its reviews, which is why I keep my own reviewer-model work on held-out material rather than published venues.
Sources
- ACL Rolling Review stats dashboard and the
acl-org/arr-healthdata repository, including the per-cycledashboard_stats.jsonthat reports active submissions and review counts (primary source for every number here). - David Jurgens, "Is the ACL Rolling Review actually broken?", on the flat-mean-under-rising-volume observation.
- Tomkins and colleagues' OpenReview analysis line, e.g. "An Open Review of OpenReview", for the reviewer-versus-area-chair question on per-review ICLR data.
- IC-Light at ICLR 2025, the straight 10/10/10/10 paper, as the contrast to ARR's averaged scale.