Dari (Afghan Persian) data annotation

What makes Dari (Afghan Persian) annotation different

دری uses the Persian alphabet, which runs right to left.

• Dari and Iranian Persian share a script and most grammar but differ in vocabulary, idiom and pronunciation. Data labelled for one is not always right for the other.
• Persian uses the zero-width non-joiner inside words and after some suffixes. Texts disagree on whether it is present, which breaks exact matching and span boundaries.
• Persian digits differ from Arabic-Indic digits in the shapes of 4, 5 and 6, and Latin digits are common as well.
• Informal writing varies widely from the formal register used in news corpora.

Normalisation checklist

Dari and Iranian Persian share most of their writing system, so mixed files are common. Settle these points first, because they change how spans and exact matches line up.

• Fix a rule for the zero-width non-joiner and tell reviewers.
• Normalise Arabic yeh and kaf to their Persian forms, or keep them, but be consistent.
• Choose one digit style.
• For span tasks, check offsets after any normalisation, because they shift when characters are removed.

A sample Dari file

Two short rows in Dari. The second row has a zero-width non-joiner inside the verb, the character that most often differs between sources.

id,text
1,امروز هوا خوب است
2,من به کابل می‌روم

Tasks people commission

• Sentiment and topic labels
• Named entities in text
• Translation quality ratings
• Transcription

Setting up a campaign

1. Add a column that says whether each row is Dari or Iranian Persian, if the file mixes them.
2. Clean the zero-width non-joiner and digits in your file before upload, not in the reviewers' answers.
3. Label a pilot of 50 rows and read the spans where reviewers differ by one character.
4. Keep the campaign private if your reviewers are colleagues, or invite Dari speakers for a crowd run.