What is Mixture of Experts?
Also called MoE, sparse model.
Mixture of experts is a model architecture that splits part of the network into many specialized sub-networks, called experts, and routes each token to only a few of them. This lets a model hold a very large total parameter count while activating a small fraction for any given token, giving the capacity of a big model at closer to the compute cost of a smaller one.
A small learned router scores the experts for each token and sends that token to the top one or two. Only the selected experts run, so the compute spent per token, sometimes reported as the active parameter count, is far below the total. Training must also balance the router, or a few popular experts absorb most traffic while the rest go unused.
Two numbers are needed to describe such a model honestly, and quoting only one is misleading. Total parameters determine memory, because every expert must be resident and available even if it is rarely selected. Active parameters determine speed and compute per token. A model can therefore be expensive to host and comparatively cheap to run per request.
Experts do not correspond to human-legible subject areas. Inspection shows routing organized around token-level and syntactic regularities rather than around topics such as law or biology, so the intuition of a panel of subject specialists is simply wrong. The specialization is real and measurable, but it is not interpretable in the terms most people expect.
The design has become common among large models because it improves the capability obtained per unit of training and serving compute. It does complicate deployment: memory requirements stay high regardless of activation sparsity, batching behavior is less predictable, and quantizing or fine-tuning a sparse model requires techniques that differ from their dense equivalents.
Key points
- Routes each token to a few sub-networks instead of the whole model.
- Total parameters set memory; active parameters set compute per token.
- Experts specialize around token patterns, not human subject areas.
- Router balancing is required or a few experts dominate traffic.
In practice
Picture a network with sixty-four experts where each token is routed to two of them. The full model may hold hundreds of billions of parameters and require all of them resident in memory, yet processing any single token touches only a few percent of that. Throughput resembles a much smaller model while the hardware bill still reflects the full count.