Consensus label
DefinitionA consensus label is the single label assigned to an item after combining the answers of several annotators. It is usually the majority answer, or a weighted answer from a model such as Dawid-Skene.
What to keep alongside it
- How many annotators answered.
- How strongly they agreed.
- Whether there was a tie.
- The individual answers, for later re-analysis.
Why not keep only the label
Low-agreement rows are often the ambiguous or mislabelled ones. Keeping the agreement figure lets you drop them, give them more review, or study them as hard cases.
How Azmaval uses it
Every exported result carries the consensus value, the number of reviewers, the agreement figure where one applies, and a tie flag.
Frequently asked questions
Is a consensus label always correct?
No. It reflects what the reviewers agreed on, and a group can agree on a wrong answer, especially if guidelines are ambiguous.
Should I train on low-agreement rows?
Often not. Filter or re-review them, or train with soft labels that keep the disagreement.