Gold standard questions
DefinitionGold standard questions are samples with a known right answer that are mixed into the normal queue. A reviewer's score on them measures accuracy without waiting for other reviewers to disagree.
How they are used
- Estimate each reviewer's accuracy early, before there are enough overlapping reviews.
- Screen out careless or automated answers.
- Check that instructions are understood, by seeing which gold rows many people miss.
Good practice
- Cover a small share of rows, enough to measure but not so many that the work feels like a test.
- Make them look like ordinary rows.
- Include real hard cases along with the obvious ones, or accuracy will look better than it is.
- Retire a gold row that many good reviewers miss. The row may be wrong.
How Azmaval uses it
In a campaign built from your data, put the right answer in a hidden column and choose a share of rows between 2 and 50 percent. Gold rows work for pick-one and rating questions in crowd campaigns. Private campaigns ignore them, since the reviewers are people you chose.
Frequently asked questions
Do reviewers know which rows are gold?
No. They look like every other row.
Can gold rows work with text answers?
Not automatically, because there is no single right text. Review a sample of free-text answers by hand.