Pashto data annotation

What makes Pashto annotation different

پښتو uses the Perso-Arabic script, which runs right to left.

• Pashto adds letters to the Perso-Arabic alphabet, such as ټ ډ ړ ږ ښ ځ څ ګ ڼ, and different keyboards give different codepoints for the same shape.
• There are two broad dialect groups, often called Southern (Kandahari) and Northern (Yusufzai or Peshawari). Say which you want, because spelling and vocabulary differ.
• Few public corpora and tools exist, so there is little to pre-label with. Expect to build training data by hand.
• Text mixes with Dari, Urdu and English words, especially on social media.

Normalisation checklist

Pashto text from the web comes in several spellings of the same word. These choices are cheap to make now and expensive to fix after reviewers have labelled.

• Pick one codepoint for the letters that have look-alikes (yeh, kaf and heh forms) and say so in the guidelines.
• Decide how to treat the zero-width non-joiner in compound words.
• Decide whether numerals are Arabic-Indic, Persian or Latin digits.
• Save files as UTF-8 and check a sample in the preview before importing.

A sample Pashto file

Two short rows in Pashto. Before you upload real text, decide whether the dialect goes in a column of its own.

id,text
1,نن هوا ښه ده
2,زه کابل ته ځم

Tasks people commission

• Speech quality rating (MOS) for synthesised voices
• Transcription of audio
• Named entities in text
• Sentiment and topic labels

Setting up a campaign

1. Decide whether you want Southern or Northern Pashto, or add a dialect column and label both.
2. Run a pilot of 50 rows with two reviewers and compare how they spell the look-alike letters.
3. Put the spelling choices in the question as a short list of examples.
4. Invite reviewers who write Pashto daily, or keep the campaign private if you will label it yourself.