Annotation guidelines
DefinitionAnnotation guidelines are the written rules annotators follow: what each label means, how to treat edge cases, and worked examples. They are the biggest single lever on label quality.
What good guidelines contain
- A definition of each label in one sentence.
- Two or three clear examples and two or three borderline ones, with the decision and why.
- Rules for the cases that cause arguments: partial objects, sarcasm, mixed languages, spelling variants.
- A version number and a changelog, so you know which rule applied to which labels.
A working loop
- Draft the guidelines with a small pilot of 50 items.
- Have two people label the pilot independently.
- Read every disagreement and fix the rule that caused it.
- Repeat until agreement is stable, then scale up.
How Azmaval uses it
Put the instructions in the question text, which reviewers see on every row. Use a comment box on the answer when you want reviewers to flag items the guidelines do not cover.
Frequently asked questions
How long should guidelines be?
As short as will do the job. Reviewers read the question on every row and a page of instructions rarely, so put the rule that matters in the question.
Should guidelines change during a campaign?
Only with a version bump and a note of which rows were labelled under which version. Changing rules silently makes earlier labels inconsistent.