Pre-training
The expensive first stage where a model learns language by predicting next tokens on internet-scale data.
Definition
Pre-training is what creates a "base model" — trillions of words of internet text fed to the network for next-token prediction. It costs millions of dollars in compute and produces a model that knows facts but doesn't know how to be helpful. Most contractor gigs are post-training (SFT + RLHF), not pre-training.
Examples
- No worker-facing tasks — this is GPU-only.
Related terms
Adapting a pretrained model to a specific task or domain by continuing training on new data.
The technical name for instruction tuning — supervised learning on (input, ideal-output) pairs.
Breaking text into chunks (tokens) so AI can process it — typically subword units, not whole words.
Looking for actual gigs that use Pre-training? Browse all live listings or filter by $100+/hr.