Pre-training

The expensive first stage where a model learns language by predicting next tokens on internet-scale data.

Definition

Pre-training is what creates a "base model" — trillions of words of internet text fed to the network for next-token prediction. It costs millions of dollars in compute and produces a model that knows facts but doesn't know how to be helpful. Most contractor gigs are post-training (SFT + RLHF), not pre-training.

Examples

  • No worker-facing tasks — this is GPU-only.

Related terms

Looking for actual gigs that use Pre-training? Browse all live listings or filter by $100+/hr.