Inter-Annotator Agreement (IAA)

How often two workers labeling the same item give the same answer — the metric that catches bad rubrics.

Definition

IAA measures whether your workforce produces consistent labels. Low IAA means the rubric is ambiguous or the workers are guessing. Common metrics: Cohen's kappa, Krippendorff's alpha. Quality projects target IAA > 0.7.

Examples

  • Two workers independently rate 100 items; IAA tells you they agreed on 87% of them

Related terms

Looking for actual gigs that use Inter-Annotator Agreement (IAA)? Browse all live listings or filter by $100+/hr.