Multi-label classification

When to use it

• Topic tags for articles and support tickets
• Several policy violations in one post
• Attributes of a product image
• Skills mentioned in a job ad

What your file looks like

The file is the same as for single choice: an id and a text column. What changes is the question type, which you set to allow several picks. A file like this is enough:

id,text
1,I was charged twice and the parcel is late
2,Please reset my password
3,The zip broke on the first day

What reviewers see

The text and a row of toggle buttons, one per option. Reviewers switch on every label that applies.

What a reviewer submits

The picks value is a bitmask of the chosen options: 3 means the first and second option. 0 means nothing applies, and it is only accepted when you allow an empty selection.

{
  "rating": {
    "picks": 3
  }
}

What you get back

value lists the labels in the most common combination. agreement is the share of reviewers who chose exactly that combination, which is stricter than per-label agreement.

{
  "n": 3,
  "value": [
    "Billing",
    "Delivery"
  ],
  "agreement": 0.667,
  "tie": false
}

Quality checks

• Agreement counts whole combinations, so a low number can mean one label is unclear. Split the question if so.
• Gold rows work for pick-one questions. For multi-label, use consensus across three reviewers.
• Allow an empty selection when many samples have no label.

Tips

• Keep the list short. Past six or seven labels reviewers skim, and recall drops.
• If a label almost always applies, drop it and record it in your data instead.
• Order options the same way every time so reviewers build a habit.

Limits

Up to 9 options. The answer is stored as a bitmask, so options keep a fixed order within a campaign.