Majority vote
DefinitionMajority vote picks the label that most annotators gave. It is the simplest way to combine several labels for one item into a single answer.
Strengths and limits
- Easy to explain and to audit.
- Needs an odd number of annotators to avoid ties, or a rule for ties.
- Treats every annotator equally, so a careless one counts as much as a careful one.
- Throws away information about how confident the group was, unless you keep the agreement figure.
Beyond it
When annotator quality varies, methods such as the Dawid-Skene model weight annotators by their estimated reliability. For marks on an image or text, a mark can be kept when a majority of annotators made a matching one.
How Azmaval uses it
Azmaval reports the winning label, the share of reviewers who gave it, and a tie flag. For spans, boxes, polygons, points and segments, a mark is kept when a majority of reviewers made a matching one.
Frequently asked questions
What happens on a tie?
The export marks the sample as a tie, so you can send it for another review instead of trusting an arbitrary pick.
How many reviewers per sample?
Three is the usual starting point for pick-one questions. Use five for hard or subjective tasks.