Speech quality rating

When to use it

• Evaluating a text-to-speech voice
• Comparing two TTS systems
• Checking recordings for a low-resource language
• Quality control for a speech dataset

What your file looks like

Each row links one audio clip and carries the reference text the speaker read, which reviewers see next to the player. A file like this is enough:

id,clip,text
1,https://example.com/audio/001.wav,سلام څنګه یاست
2,https://example.com/audio/002.wav,نن هوا ښه ده

What reviewers see

An audio player, the reference text if you supply it, and the rating buttons. The screen reads right to left for Pashto, Persian, Urdu and Arabic.

What a reviewer submits

One score per clip. The clip link is stored with the sample.

{
  "rating": {
    "score": 5
  }
}

What you get back

Average the value column over a system's clips to get its MOS. The export also lists every reviewer's score, so you can compute confidence intervals.

{
  "n": 5,
  "value": 3.8,
  "agreement": null,
  "tie": false
}

Quality checks

• Quality rules flag answers submitted too fast and reviewers who answer too many clips an hour.
• Gold checks put a clip with a known score in the queue and flag reviewers who rate it far off.
• Reviewer tiers are based on measured accuracy, not claimed fluency.

Tips

• Include natural recordings as anchors, so listeners have a reference for the top of the scale.
• Randomise which system a clip comes from. Order effects are real.
• Ten or more ratings per system and clip pair gives a stable mean.

Limits

Audio is linked by URL. Use WAV or MP3. Long clips slow reviewers down, so keep them under about ten seconds for naturalness ratings.