Cohen's kappa
DefinitionCohen's kappa is a statistic for agreement between two annotators that corrects for the agreement expected by chance. It equals (po - pe) / (1 - pe), where po is observed agreement and pe is chance agreement.
Formula and example
Suppose two annotators label 100 items and agree on 80, so po = 0.80. From how often each used each label, chance agreement works out to pe = 0.50. Then kappa = (0.80 - 0.50) / (1 - 0.50) = 0.60.
Limits
- It compares exactly two annotators. For more, use Fleiss' kappa or Krippendorff's alpha.
- It treats all disagreements the same. A rating of 1 against 5 counts the same as 1 against 2, so for ordered scales a weighted kappa or alpha fits better.
- It can look low when one label is very common, even with high raw agreement.
Origin
Jacob Cohen introduced it in 1960 in a paper on a coefficient of agreement for nominal scales.
How Azmaval uses it
Azmaval exports each reviewer's answer under a pseudonym per project, so you can compute Cohen's kappa for any pair of reviewers yourself.
Frequently asked questions
Can kappa be negative?
Yes. A negative value means agreement is lower than chance, which usually signals confusion about the labels or a swapped option order.
Is there an official scale for kappa values?
No. The Landis and Koch labels (0.61 to 0.80 substantial, above 0.80 almost perfect) are widely quoted but are a convention.