Attention check
DefinitionAn attention check is a task with an obvious answer, placed among normal tasks, to catch annotators who are not reading or who are submitting automated answers.
Examples
- An audio clip of silence that must be rated at the bottom of the scale.
- A text that says Select option 3.
- A question whose answer is plainly visible in the sample.
Limits
Attention checks catch gross inattention, not subtle errors. They are most useful in open crowds and less useful in small groups of people you chose. Do not make them so frequent that honest reviewers feel distrusted.
How Azmaval uses it
Built-in task types such as speech quality rating can carry attention checks, and campaigns built from your data use gold rows for the same purpose. Quality rules also flag answers submitted too quickly and reviewers who answer too many in an hour.
Frequently asked questions
What happens when a reviewer fails one?
The failure counts against their accuracy and can flag them for review. During probation, a low measured accuracy can lead to suspension.
Is an attention check the same as a gold question?
They are close. A gold question has a known answer on real content, while an attention check is deliberately trivial.