How many reviewers should label each item?
Three reviewers per item is the usual starting point: with reviewers who are each right 80 percent of the time, a majority of three is right about 90 percent of the time, and five reaches about 94 percent. The sums assume reviewers make independent mistakes.
The arithmetic
Suppose each reviewer is right with probability p, and mistakes are independent. A majority of three is right when at least two are right: p cubed plus three times p squared times (1 - p). For p = 0.8 that is 0.512 + 0.384 = 0.896. A majority of five is right when at least three are right, which for p = 0.8 comes to 0.942.
| Reviewer accuracy | 1 reviewer | 3 reviewers | 5 reviewers |
|---|---|---|---|
| 0.90 | 0.900 | 0.972 | 0.991 |
| 0.80 | 0.800 | 0.896 | 0.942 |
| 0.70 | 0.700 | 0.784 | 0.837 |
| 0.60 | 0.600 | 0.648 | 0.683 |
What the table says
- Going from one reviewer to three is the big step, worth about ten points at 80 percent accuracy.
- Going from three to five adds about five points. Spend it on the rows that are hard, not on every row.
- When reviewers are only slightly better than chance, extra reviewers help little. Fix the guidelines instead.
Where the sums break
Independence is the weak assumption. If your instructions are ambiguous, reviewers make the same mistake together, and a majority is wrong as often as one person. Agreement then looks high while accuracy is low. The same happens when everyone starts from the same model suggestion.
A practical plan
- Start with three reviewers per item on a pilot of 50 to 100 rows.
- Read the rows where reviewers disagree. Fix the guideline that caused each one.
- Add gold rows with known answers to measure each reviewer's accuracy directly.
- Use three reviewers for the main run, and send low-agreement rows to two more.
Ties and even numbers
With two or four reviewers a tie is possible. A tie is information: send those rows to another reviewer rather than picking one at random.
Frequently asked questions
Is one reviewer ever enough?
Yes, for a private dataset you label yourself, or when you will review the results anyway. For data that trains a model, use at least three on the rows that matter.
Does this apply to ratings and boxes?
The principle does. For ratings, you average instead of voting. For boxes and spans, you keep marks a majority drew.