Yes or no questions

When to use it

• Is this transcript correct?
• Does this answer match the question?
• Is this image safe to publish?
• Is this model label right? (confirm or reject a suggestion)

What your file looks like

Put the statement to check in one column. The question and the two answers come from the form, so the file needs no label column. A file like this is enough:

id,claim
1,The bridge opened in 1932
2,The river is the longest in the country
3,The museum is closed on Mondays

What reviewers see

The claim and two buttons. Pressing Y or N answers and moves on, so a reviewer can work through rows without the mouse.

What a reviewer submits

Yes is option 1 and No is option 2. A yes or no question is an ordinary pick-one question with two options, so the stored answer and the export look the same.

{
  "rating": {
    "choice": 1
  }
}

What you get back

agreement of 1 means all reviewers said the same. With an even number of reviewers a tie is possible, so use three or five.

{
  "n": 3,
  "value": "Yes",
  "agreement": 1,
  "tie": false
}

Quality checks

• Hide a known answer in a column and use it as a gold check.
• Use an odd number of reviews per sample so there are no ties.
• Add a comment box so reviewers can say why they rejected a row.

Tips

• Phrase the question so Yes is the common case. It speeds reviewers up and makes mistakes easier to spot.
• Avoid double negatives and compound claims. One fact per question.
• For a model's output, show the model's guess as a suggestion and measure how often reviewers overrule it.

Limits

Exactly two options. Everything else about pick-one questions applies, including comments and gold rows.