Pairwise preference

When to use it

• Comparing two model replies (RLHF-style preference data)
• A/B tests of translations
• Choosing between two TTS voices
• Ranking search results

What your file looks like

Use one column for the prompt and one for each candidate reply. Mix up which system fills reply_a so no system always appears first. A file like this is enough:

id,prompt,reply_a,reply_b
1,Explain inflation in one sentence,Prices rise over time.,Inflation is the rate at which the general price level of goods and services rises.
2,Name a prime number,Seven,Nine

What reviewers see

The prompt and both replies on one screen, with one button for each reply and one for a tie.

What a reviewer submits

A preference is a pick-one question whose options are the two candidates and, if you want it, a tie.

{
  "rating": {
    "choice": 2
  }
}

What you get back

value is the winner by majority. Randomise which candidate is A in your file, otherwise position bias leaks into your labels.

{
  "n": 3,
  "value": "Reply B",
  "agreement": 0.667,
  "tie": false
}

Quality checks

• Swap A and B on a share of rows and check that preferences flip, which reveals position bias.
• Use gold rows where one reply is clearly broken.
• Keep a comment box for the reason when training a reward model needs rationales.

Tips

• Add an Equal option. Forcing a choice on a draw adds noise.
• Show only the text that matters. Remove model names and metadata from the shown columns.
• Three reviewers per pair is the usual minimum for stable majorities.

Limits

Two candidates in two columns, plus the options you define. Longer text is fine; very long replies are easier to compare in separate questions.