Urdu data annotation

What makes Urdu annotation different

اردو uses a Perso-Arabic script, usually displayed in Nastaliq style, which runs right to left.

• Romanised Urdu is common in messages and social media, and spelling is not standardised, so the same word has many spellings.
• Text often mixes Urdu with English, and with Hindi vocabulary that is spelled differently.
• Urdu adds letters such as ٹ ڈ ڑ ں ھ ے to the Arabic alphabet.
• The Nastaliq style that readers expect is a display choice. Check that your font renders it before reviewers see it.

Normalisation checklist

Urdu data often arrives in two scripts and many spellings. These points decide whether your labels stay consistent across them.

• Say whether romanised text is in scope.
• Fix one codepoint for look-alike letters, such as the two heh forms.
• Decide how to label code-mixed sentences.
• Keep the text in logical order in your file, not visual order.

A sample Urdu file

The same sentence in Urdu script and in romanised form. Say in the question whether both kinds of row belong in the campaign.

id,text
1,آج موسم بہت اچھا ہے
2,aaj mausam bohat acha hai

Tasks people commission

• Sentiment and topic labels
• Named entities in text
• Speech quality rating
• Pairwise preference for model replies

Setting up a campaign

1. Split romanised and native-script rows into two files if you want separate results for each.
2. Pick a font that draws Nastaliq well and check it in the preview.
3. Show reviewers two worked examples of code-mixed sentences and how to label them.
4. Invite Urdu readers for the crowd run, or label the data yourself in a private campaign.