Arabic data annotation

What makes Arabic annotation different

العربية uses the Arabic script, which runs right to left.

• Modern Standard Arabic is used in writing and news. Egyptian, Gulf, Levantine and Maghrebi dialects dominate speech and social media, and they differ a lot.
• Short vowel marks are usually left out, so words are ambiguous without context.
• Letter variants, such as the forms of alef and the endings ta marbuta and ha, are inconsistent in real text.
• A stretching character (tatweel) is sometimes inserted for emphasis and breaks exact matching.

Normalisation checklist

Arabic spelling varies with region, keyboard and habit. Decide these points once so that variants do not show up as label disagreement.

• Decide whether to normalise alef forms and the ya and alef maqsura endings.
• Strip tatweel, or keep it.
• Decide how to handle diacritics.
• Record the dialect in a column so you can filter later.

A sample Arabic file

Two short rows in Modern Standard Arabic. Add a dialect column when the file mixes dialects, so you can filter results later.

id,text
1,الطقس جميل اليوم
2,سافرت إلى القاهرة أمس

Tasks people commission

• Dialect identification
• Named entities in text
• Preference data for model replies
• Speech transcription

Setting up a campaign

1. Add a dialect column when the file mixes varieties, and run a separate campaign for each if the labels differ.
2. Strip tatweel and unify alef forms in the file before upload if you want exact matching.
3. Check that diacritics survive the upload by looking at the preview.
4. Invite reviewers who know the dialect you are labelling, since a Gulf speaker may misread Maghrebi text.