Data annotation glossary
Each entry starts with a one or two sentence definition, then gives a formula or example, the limits of the method, and how Azmaval applies it. Read them in any order.
Inter-annotator agreement
Inter-annotator agreement measures how often independent annotators give the same label to the same item.
Cohen's kappa
Cohen's kappa is a statistic for agreement between two annotators that corrects for the agreement expected by chance.
Fleiss' kappa
Fleiss' kappa extends chance-corrected agreement to more than two annotators.
Krippendorff's alpha
Krippendorff's alpha is an agreement coefficient that works with any number of annotators, missing labels and different scales: nominal, ordinal, interval and ratio.
Dawid-Skene model
The Dawid-Skene model estimates the true label of each item and the error pattern of each annotator together, using the expectation-maximisation algorithm.
Gold standard questions
Gold standard questions are samples with a known right answer that are mixed into the normal queue.
Majority vote
Majority vote picks the label that most annotators gave.
Consensus label
A consensus label is the single label assigned to an item after combining the answers of several annotators.
Attention check
An attention check is a task with an obvious answer, placed among normal tasks, to catch annotators who are not reading or who are submitting automated answers.
Mean opinion score (MOS)
A mean opinion score is the average of listeners' ratings of quality, normally on a five-point scale from bad to excellent.
Intersection over union (IoU)
Intersection over union measures how much two shapes overlap: the area they share divided by the area they cover together.
BIO tagging
BIO tagging labels each token with B for the Beginning of an entity, I for Inside an entity, or O for Outside any entity.
Named entity recognition (NER)
Named entity recognition finds mentions of real-world things in text, such as people, places and organisations, and labels each with its type.
Pre-annotation
Pre-annotation means a model fills in a first-guess label and humans confirm or correct it.
Annotation guidelines
Annotation guidelines are the written rules annotators follow: what each label means, how to treat edge cases, and worked examples.
Wilson score interval
The Wilson score interval is a confidence interval for a proportion, such as a reviewer's accuracy.
Position bias
Position bias is the tendency to prefer an option because of where it is shown, such as the first answer in a pair, rather than because it is better.