Distillation

Training a smaller model to mimic a larger one — captures most of the capability at a fraction of the cost.

Definition

Distillation transfers knowledge from a large "teacher" model to a small "student" by training the student on the teacher's outputs. Most production AI uses distilled models. Worker tasks can include rating teacher outputs that get used as training targets.

Examples

  • GPT-4 outputs distilled into a 7B-parameter model

Related terms

Looking for actual gigs that use Distillation? Browse all live listings or filter by $100+/hr.