Krippendorff's alpha
DefinitionKrippendorff's alpha is an agreement coefficient that works with any number of annotators, missing labels and different scales: nominal, ordinal, interval and ratio. It equals 1 minus the ratio of observed to expected disagreement.
Formula
alpha = 1 - Do / De, where Do is the observed disagreement among labels given to the same item and De is the disagreement expected if labels were assigned at random with the same overall frequencies. What counts as disagreement depends on the scale, so a 1 against a 5 weighs more than a 1 against a 2 on an interval scale.
Reading the value
Klaus Krippendorff suggests alpha of at least 0.80 for reliable conclusions and 0.667 as the lowest value for tentative ones. A value of 0 means agreement at chance level.
Why teams like it
- It does not need every annotator to label every item.
- It handles ratings as numbers as well as categories.
- One coefficient works across the whole project.
How Azmaval uses it
The Azmaval quality report includes Krippendorff's alpha for interval data, computed from reviewers' ratings on each sample.
Frequently asked questions
When should I use alpha instead of kappa?
When annotators are not the same across items, when labels are missing, or when the labels are ordered or numeric.
What is the alpha threshold?
0.80 or above for reliable data, and 0.667 to 0.80 for tentative conclusions, according to Krippendorff's own guidance.