Wilson score interval
DefinitionThe Wilson score interval is a confidence interval for a proportion, such as a reviewer's accuracy. Unlike the simple normal interval it behaves well with few trials and with proportions near 0 or 1.
Why it fits reviewer accuracy
A reviewer who got 4 of 4 gold questions right has a raw accuracy of 100 percent, but four answers prove little. The Wilson lower bound for 4 of 4 at 95 percent confidence is about 0.51, which is a fairer picture of what we know. With 40 of 40 it rises to about 0.91.
Formula
With observed proportion p, n trials and z = 1.96 for 95 percent confidence, the interval is centred at (p + z squared / 2n) / (1 + z squared / n), with half-width z / (1 + z squared / n) times the square root of p(1 - p)/n + z squared / 4n squared.
How Azmaval uses it
Reviewer tiers and probation decisions use Wilson bounds: a high tier needs a high lower bound, and a reviewer is suspended only when the upper bound of their accuracy is low. Small samples therefore neither promote nor punish anyone early.
Frequently asked questions
Why not just use accuracy?
Raw accuracy from a handful of answers is unreliable. The interval says how much to trust it.
Is it the same as a Bayesian credible interval?
No, though for a flat prior the two are close. The Wilson interval is a frequentist method with a simple closed form.