Inter-annotator agreement
DefinitionInter-annotator agreement measures how often independent annotators give the same label to the same item. It is the main evidence that a labelling task is clear and that the resulting data can be trusted.
Why it matters
If two careful people disagree about a label, a model trained on those labels learns the disagreement as noise. Agreement tells you whether your guidelines are clear enough, whether a label set is too fine, and how good a model can possibly get, since a model rarely beats the humans' own consistency.
How it is measured
- Percent agreement: the share of items with identical labels. Easy to read, but inflated when one label is very common.
- Cohen's kappa: for two annotators, corrects percent agreement for chance.
- Fleiss' kappa: the same correction for more than two annotators who each label every item.
- Krippendorff's alpha: works with any number of annotators, missing labels and several kinds of scale.
Reading the number
Coefficients run from below 0 (worse than chance) through 0 (chance) to 1 (perfect). Common rules of thumb call 0.8 and above good and 0.67 to 0.8 tentative, but the right bar depends on the cost of an error in your use case.
How Azmaval uses it
Azmaval records every reviewer's own answer and reports agreement per sample (the share of reviewers who gave the winning answer). The quality report includes Krippendorff's alpha for rating data.
Frequently asked questions
What is a good level of agreement?
For kappa and alpha, 0.8 or higher is usually treated as reliable and 0.67 to 0.8 as acceptable for tentative conclusions. Subjective tasks such as naturalness ratings often reach lower values.
How many annotators do I need to measure it?
Two is enough for Cohen's kappa. For stable estimates on a campaign, three reviewers per sample is a common minimum.