Single-choice classification

When to use it

• Sentiment or topic labels for text
• Spam, toxicity or safety triage
• Choosing the best of several model outputs
• Checking a label that already exists

What your file looks like

One text column is enough. The id column stays hidden, and you write the options and the question in the form. A file like this is enough:

id,text
1,The delivery was early and the box was undamaged
2,Nothing arrived and nobody answered my emails
3,It works as described

What reviewers see

The text, the question, and one large button per option. A two-option question gets Y and N shortcuts, and you can add a comment box next to the answer.

What a reviewer submits

The choice is the 1-based position of the option the reviewer picked. 0 means none of these, and it is only accepted when you allow it.

{
  "rating": {
    "choice": 3
  }
}

What you get back

value is the winning label. agreement is the share of reviewers who gave that answer, so 0.667 means two of three. tie is true when two answers got the same number of votes.

{
  "n": 3,
  "value": "Negative",
  "agreement": 0.667,
  "tie": false
}

Quality checks

• Plant gold rows: put the right answer in a hidden column and Azmaval tests reviewers on a share of rows (crowd campaigns only).
• Ask for three reviews per sample and read agreement to find the confusing rows.
• Show a suggestion from a model and let reviewers confirm or change it.

Tips

• Keep options mutually exclusive. If reviewers keep disagreeing, two options probably overlap.
• Allow none of these when real data can fall outside your list, so reviewers do not guess.
• Put the instructions in the question, not in a separate document nobody opens.

Limits

Up to 9 options. Options can be fixed for the campaign or come from a column, so every row can have its own list.