Distillation
Training a smaller model to mimic a larger one — captures most of the capability at a fraction of the cost.
Definition
Distillation transfers knowledge from a large "teacher" model to a small "student" by training the student on the teacher's outputs. Most production AI uses distilled models. Worker tasks can include rating teacher outputs that get used as training targets.
Examples
- GPT-4 outputs distilled into a 7B-parameter model
Related terms
Looking for actual gigs that use Distillation? Browse all live listings or filter by $100+/hr.