Inter-Annotator Agreement (IAA)
How often two workers labeling the same item give the same answer — the metric that catches bad rubrics.
Definition
IAA measures whether your workforce produces consistent labels. Low IAA means the rubric is ambiguous or the workers are guessing. Common metrics: Cohen's kappa, Krippendorff's alpha. Quality projects target IAA > 0.7.
Examples
- Two workers independently rate 100 items; IAA tells you they agreed on 87% of them
Related terms
Rubric
The structured scoring guide workers use to evaluate AI outputs consistently.
Gold Label / Gold Standard
A reference answer treated as definitively correct — used to grade workers and AI alike.
Evaluation (Eval)
Testing AI outputs against quality criteria — the largest single category of AI training work.
Looking for actual gigs that use Inter-Annotator Agreement (IAA)? Browse all live listings or filter by $100+/hr.