Mixture of Experts (MoE)

A model architecture where different sub-networks specialize in different inputs — gets you GPT-4 quality at smaller compute.

Definition

MoE models have many "expert" sub-networks but only activate a few per input. Mixtral, GPT-4, Llama 4 use MoE. From a worker view, MoE is invisible — you still see prompts and responses.

Examples

  • Mixtral 8x22B
  • Llama 4 Behemoth
  • GPT-4 (per leaks)

Related terms

Looking for actual gigs that use Mixture of Experts (MoE)? Browse all live listings or filter by $100+/hr.