# Azmaval

> Open data annotation: turn a CSV into a labelling screen and get it reviewed by vetted people.

Azmaval builds a labelling screen from a CSV or JSON file and gets it reviewed by vetted people, with consensus, gold checks and agreement figures. It is in early access and invitation only. Pages below are plain text and kept current; each states the date it was last checked.

## Annotation types

- [Single choice](https://azmaval.com/annotation/single-choice-classification): Each sample gets exactly one label from a short list: sentiment, topic, spam or not, the right translation.
- [Multi-label](https://azmaval.com/annotation/multi-label-classification): Each sample can carry several labels at once: every topic in an article, every defect in a photo.
- [Yes or no](https://azmaval.com/annotation/yes-no-questions): A two-option question with keyboard shortcuts. The fastest way to review, filter or verify at volume.
- [Rating scales](https://azmaval.com/annotation/rating-scales): Reviewers rate each sample on a numeric scale, such as 1 to 5 for fluency, relevance or helpfulness.
- [Speech quality (MOS)](https://azmaval.com/annotation/speech-quality-rating): Listeners rate an audio clip, usually for naturalness on a 1 to 5 scale. The result is a Mean Opinion Score.
- [Pairwise preference](https://azmaval.com/annotation/pairwise-preference): Reviewers see two options for the same input and pick the better one. The standard format for preference data.
- [Free text](https://azmaval.com/annotation/free-text-answers): Reviewers write an answer: a transcript, a translation, a short explanation. Every reviewer's text comes back.
- [Text spans (NER)](https://azmaval.com/annotation/text-span-labelling): Reviewers select parts of a text and give each a label, such as person, place or organisation. The basis of named entity recognition data.
- [Relations](https://azmaval.com/annotation/relation-extraction): After marking spans, reviewers link two of them with a named relation, such as works for or capital of.
- [Bounding boxes](https://azmaval.com/annotation/bounding-boxes): Reviewers draw rectangles around objects in an image and label each one. The base format for object detection data.
- [Polygons](https://azmaval.com/annotation/polygon-annotation): Reviewers click around the outline of an object to make a polygon. Tighter than a box, and the base for segmentation masks.
- [Keypoints](https://azmaval.com/annotation/keypoint-annotation): Reviewers click to place labelled points on an image: eyes, joints, corners, landmarks.
- [Audio and video segments](https://azmaval.com/annotation/audio-segments): Reviewers play a clip and mark where something starts and ends, with a label for each stretch.
- [All annotation types](https://azmaval.com/annotation)

## File formats

- [CSV](https://azmaval.com/formats/csv): Paste or upload a CSV and Azmaval reads the header, guesses each column's type and builds the review screen. CSV is also an export format.
- [TSV](https://azmaval.com/formats/tsv): Tab, semicolon and pipe separated files import the same way as CSV. The delimiter is detected, so you do not choose it.
- [JSON Lines](https://azmaval.com/formats/jsonl): One JSON object per line. Import keeps your fields as columns, and export gives one result record per sample, ready for a training pipeline.
- [JSON](https://azmaval.com/formats/json): A JSON file that is a list of objects imports like a table. Keys are columns. Label Studio exports are recognised automatically.
- [Label Studio JSON](https://azmaval.com/formats/label-studio-json): Drop in a Label Studio export and get a working campaign with the questions rebuilt. Export results back in Label Studio's task format.
- [CoNLL (BIO)](https://azmaval.com/formats/conll): Campaigns with span questions export in CoNLL format: one token per line with a BIO tag, the format most NER training code reads.
- [Parquet](https://azmaval.com/formats/parquet): A columnar export for analysis in pandas, DuckDB or Spark. The consensus result and its agreement are separate typed columns.
- [All file formats](https://azmaval.com/formats)

## Compare

- [Azmaval vs Label Studio](https://azmaval.com/compare/label-studio): Label Studio is a flexible open-source labelling tool you run yourself. Azmaval is a hosted service that builds the screen from your data and brings reviewers and quality checks with it.
- [Azmaval vs Argilla](https://azmaval.com/compare/argilla): Argilla is an open-source tool for building and reviewing datasets, strong for NLP and language model feedback. Azmaval adds a vetted reviewer network, quality gating and image, audio and video tasks.
- [Azmaval vs Scale AI](https://azmaval.com/compare/scale-ai): Scale AI sells managed data labelling and evaluation to enterprises, with pricing by quote. Azmaval is self-service: you bring a file, set the pay and the review rules, and see the results.
- [All compare](https://azmaval.com/compare)

## Glossary

- [Inter-annotator agreement](https://azmaval.com/glossary/inter-annotator-agreement): Inter-annotator agreement measures how often independent annotators give the same label to the same item.
- [Cohen's kappa](https://azmaval.com/glossary/cohens-kappa): Cohen's kappa is a statistic for agreement between two annotators that corrects for the agreement expected by chance.
- [Fleiss' kappa](https://azmaval.com/glossary/fleiss-kappa): Fleiss' kappa extends chance-corrected agreement to more than two annotators.
- [Krippendorff's alpha](https://azmaval.com/glossary/krippendorffs-alpha): Krippendorff's alpha is an agreement coefficient that works with any number of annotators, missing labels and different scales: nominal, ordinal, interval and ratio.
- [Dawid-Skene model](https://azmaval.com/glossary/dawid-skene-model): The Dawid-Skene model estimates the true label of each item and the error pattern of each annotator together, using the expectation-maximisation algorithm.
- [Gold standard questions](https://azmaval.com/glossary/gold-standard-questions): Gold standard questions are samples with a known right answer that are mixed into the normal queue.
- [Majority vote](https://azmaval.com/glossary/majority-vote): Majority vote picks the label that most annotators gave.
- [Consensus label](https://azmaval.com/glossary/consensus-label): A consensus label is the single label assigned to an item after combining the answers of several annotators.
- [Attention check](https://azmaval.com/glossary/attention-check): An attention check is a task with an obvious answer, placed among normal tasks, to catch annotators who are not reading or who are submitting automated answers.
- [Mean opinion score (MOS)](https://azmaval.com/glossary/mean-opinion-score): A mean opinion score is the average of listeners' ratings of quality, normally on a five-point scale from bad to excellent.
- [Intersection over union (IoU)](https://azmaval.com/glossary/intersection-over-union): Intersection over union measures how much two shapes overlap: the area they share divided by the area they cover together.
- [BIO tagging](https://azmaval.com/glossary/bio-tagging): BIO tagging labels each token with B for the Beginning of an entity, I for Inside an entity, or O for Outside any entity.
- [Named entity recognition (NER)](https://azmaval.com/glossary/named-entity-recognition): Named entity recognition finds mentions of real-world things in text, such as people, places and organisations, and labels each with its type.
- [Pre-annotation](https://azmaval.com/glossary/pre-annotation): Pre-annotation means a model fills in a first-guess label and humans confirm or correct it.
- [Annotation guidelines](https://azmaval.com/glossary/annotation-guidelines): Annotation guidelines are the written rules annotators follow: what each label means, how to treat edge cases, and worked examples.
- [Wilson score interval](https://azmaval.com/glossary/wilson-score-interval): The Wilson score interval is a confidence interval for a proportion, such as a reviewer's accuracy.
- [Position bias](https://azmaval.com/glossary/position-bias): Position bias is the tendency to prefer an option because of where it is shown, such as the first answer in a pair, rather than because it is better.
- [All glossary](https://azmaval.com/glossary)

## Languages

- [Pashto](https://azmaval.com/languages/pashto): Pashto is an official language of Afghanistan and is also spoken in Pakistan. Annotation data for it is scarce, so careful guidelines and native reviewers matter more than volume.
- [Dari (Afghan Persian)](https://azmaval.com/languages/dari): Dari is the Afghan variety of Persian and an official language of Afghanistan. Persian data from Iran differs in vocabulary and pronunciation, so label Dari data with Dari speakers.
- [Urdu](https://azmaval.com/languages/urdu): Urdu is written in a Perso-Arabic script and widely used online in both the native script and in romanised form. Decide which forms your data should cover before labelling.
- [Arabic](https://azmaval.com/languages/arabic): Arabic data comes in Modern Standard Arabic and many regional dialects. Say which one you label, and normalise letter variants before reviewers see the text.
- [All languages](https://azmaval.com/languages)

## Blog

- [How many reviewers per item?](https://azmaval.com/blog/how-many-reviewers-per-item): Three reviewers beat one by a wide margin, and five add less than you expect. The arithmetic, and when it stops holding.
- [From a spreadsheet to a labelling screen](https://azmaval.com/blog/from-spreadsheet-to-labelling-screen): Paste a CSV, check the preview, and get a review screen built from your columns. What the tool guesses, and what to check.
- [All blog](https://azmaval.com/blog)

## Optional

- [Full text of every page](https://azmaval.com/llms-full.txt)
- [Sitemap](https://azmaval.com/sitemap.xml)
- [Blog feed](https://azmaval.com/blog/rss.xml)