Fleiss' kappa
DefinitionFleiss' kappa extends chance-corrected agreement to more than two annotators. It needs every item to be labelled by the same number of annotators, and it treats the labels as unordered categories.
When to use it
Use it for category labels such as sentiment or topic when each sample gets the same number of reviews. If some items have fewer labels, or the labels are ordered numbers, Krippendorff's alpha is the better fit.
How it works
For each item it computes the share of annotator pairs that agree. It averages that over items and corrects for the agreement expected if annotators picked labels at random in proportion to each label's overall frequency.
Origin
Joseph Fleiss published it in 1971 as a measure of nominal scale agreement among many raters.
How Azmaval uses it
A campaign that asks for the same number of reviews on every sample produces exactly the table Fleiss' kappa needs, and the JSONL export gives you each reviewer's answers to build it.
Frequently asked questions
How is it different from Cohen's kappa?
Cohen's kappa compares two named annotators. Fleiss' kappa handles any fixed number of annotators and does not track who gave which label.
Does it work if annotators differ between items?
Yes, as long as each item has the same number of ratings. The annotators do not have to be the same people.