Position bias
DefinitionPosition bias is the tendency to prefer an option because of where it is shown, such as the first answer in a pair, rather than because it is better.
Where it appears
- Side-by-side comparisons of two model replies.
- Long lists of options.
- Rating tasks, where the first clips in a session shape later ratings.
How to control it
- Randomise the position of each candidate in your file.
- Show a share of pairs twice with positions swapped, and measure how often the preference flips.
- Add an Equal option so a draw is not forced into a pick.
- Report preference rates with and without the swapped pairs.
How Azmaval uses it
Azmaval shows options in the order of your file, so randomise or swap in the data you upload. A pairwise campaign with swapped duplicates gives you a direct measure of the bias.
Frequently asked questions
Do language models show position bias too?
Yes. When models are used as judges, they often favour the first or longest answer, so the same swapping check applies.
How much swapping is enough?
A tenth to a quarter of pairs, shown twice, gives a stable estimate.