Mean opinion score (MOS)
DefinitionA mean opinion score is the average of listeners' ratings of quality, normally on a five-point scale from bad to excellent. It is the standard way to compare speech synthesis and speech coding systems.
How a MOS test runs
- Pick the clips and the systems to compare.
- Present each clip to several listeners in random order.
- Ask one question, such as how natural does this speech sound, on a 1 to 5 scale.
- Average the ratings for each system, and report a confidence interval.
Design points
- Include a natural recording as an anchor for the top of the scale.
- Randomise order, since listeners drift as they go.
- Collect enough ratings per system to get a narrow interval, often hundreds of ratings.
- State the language and the listener population. A MOS from non-native listeners is not comparable with one from native listeners.
Standard
The absolute category rating method is described in ITU-T Recommendation P.800.
How Azmaval uses it
Azmaval ships a speech quality rating task for MOS studies, with right-to-left screens and reviewer pools for Pashto, Dari, Urdu and Arabic.
Frequently asked questions
Is a MOS of 4 good?
On a five-point scale, 4 is usually good, and natural recordings often land around 4.3 to 4.7. Compare against an anchor in the same test, not against numbers from other tests.
How many listeners do I need?
Enough ratings per system to make the confidence interval narrower than the difference you care about. Ten or more per clip is a common target.