Dawid-Skene model
DefinitionThe Dawid-Skene model estimates the true label of each item and the error pattern of each annotator together, using the expectation-maximisation algorithm. Reliable annotators then count for more than unreliable ones.
How it works
Each annotator gets a confusion matrix: the chance that they give label j when the true label is i. The algorithm alternates between guessing true labels from the current matrices and re-estimating the matrices from the guessed labels, until the numbers settle.
Compared with majority vote
Majority vote gives every annotator one equal vote. Dawid-Skene learns that one annotator is almost always right and another answers at random, and weights them accordingly. The gain is largest when annotator quality varies a lot, and small when everyone is similar.
Origin
A. P. Dawid and A. M. Skene published it in 1979 in Applied Statistics, in a paper on maximum likelihood estimation of observer error rates.
How Azmaval uses it
Azmaval uses Dawid-Skene on pick-one labels to estimate how accurate each reviewer is against the weighted consensus. Reviewer tiers use that accuracy, with a statistical lower bound so that a small sample does not earn a high tier.
Frequently asked questions
Does Dawid-Skene need gold labels?
No. It works from the annotators' answers alone, although known answers improve the estimates.
How many annotators does it need per item?
At least two, and more is better. With one label per item there is nothing to compare.