Rating scales

When to use it

• Fluency and adequacy of machine translation
• Helpfulness of a model reply
• Relevance of a search result
• Any 1 to 5 quality score

What your file looks like

Include every field the rater needs to judge, here the prompt and the reply. Both columns are shown, and the scale is set in the form. A file like this is enough:

id,prompt,reply
1,How do I reset my router?,Hold the reset button for ten seconds then wait for the lights.
2,What is a mortgage?,A mortgage is a loan used to buy property.

What reviewers see

The fields you chose to show, the question, and a row of numbered buttons with your end labels under the first and last.

What a reviewer submits

The score is the number the reviewer picked, inside the min and max you set.

{
  "rating": {
    "score": 4
  }
}

What you get back

value is the mean of all scores, rounded to two decimals. Scales have no agreement figure in the export, so check the spread by comparing the raw reviews.

{
  "n": 3,
  "value": 4.33,
  "agreement": null,
  "tie": false
}

Quality checks

• Define every point of the scale in the question or the end labels, so a 3 means the same to everyone.
• Use three or more reviewers and average, because individual raters differ in how generously they score.
• Add gold rows with an obvious 1 or 5 to catch inattentive reviewers.

Tips

• Five points is a good default. Wider scales add noise more often than information.
• Rate one thing at a time. Two axes on one screen need two questions.
• For speech quality see the Mean Opinion Score page, which uses the same mechanism with audio.

Limits

A scale runs between a minimum and a maximum you set, and the end labels are optional.