Pre-annotation
DefinitionPre-annotation means a model fills in a first-guess label and humans confirm or correct it. It makes labelling faster, at the cost of a risk that reviewers accept wrong guesses.
Benefits
- Faster work for tasks where the model is already decent.
- A direct measure of how often people overrule the model.
- Hard cases surface where reviewers change the guess.
Risks
- Anchoring: people tend to accept what they are shown.
- Errors in the model are copied into the data.
- Agreement looks higher than it is, since all reviewers start from the same guess.
Mitigations
Hold back a share of rows with no suggestion and compare accuracy. Plant gold rows. Show suggestions only for tasks where the model beats a chosen accuracy.
How Azmaval uses it
A hidden column can pre-fill a suggestion on any question. Reviewers see the suggested answer already selected, with a note that it is suggested, and can change it.
Frequently asked questions
Does the suggestion appear in the export?
Reviewers' final answers are exported. Your suggestion stays in your own column.
Can I use model uncertainty to order the queue?
Yes. A hidden number column can set the order, highest first, which is a simple form of active learning.