The transformer architecture that launched the modern AI era is deceptively unchanged. What has changed underneath is how computation is routed through it. Mixture-of-experts (MoE) designs have moved from a research sideline to the default at frontier scale.
What MoE actually is
A mixture-of-experts layer replaces one large feedforward block with many smaller ones plus a router that picks which few to activate for each token. A model may hold hundreds of billions of parameters in aggregate but activate only a fraction on any given forward pass. Total capacity grows without a proportional increase in compute per token.
Why now
The theoretical case for MoE was always strong. What changed is the plumbing. Training MoE at scale requires careful load balancing, sophisticated all-to-all communication, and expert-parallel sharding schemes that only recently became practical. The hardware and framework work of 2023–24 quietly unblocked what had been an implementation nightmare.
The economic case is also stronger. Serving inference is now a bigger cost line than training at most large deployments. An MoE model that activates 30B of 300B parameters costs closer to a 30B dense model to serve while performing more like a 200B one. That ratio is why every major lab now ships MoE variants.
- Total parameters increase capacity; active parameters determine cost per token.
- Routing quality is now the dominant research question — bad routing wastes capacity.
- Distillation from MoE teachers into smaller dense students is emerging as a standard recipe.
The routing problem
Getting a router to send the right tokens to the right experts is harder than it looks. Early MoE work collapsed into using only a handful of experts consistently; balancing losses were introduced to force diversity. Modern designs use soft routing, expert-choice routing, or hierarchical schemes to distribute load without sacrificing specialization.
Fine-tuning MoE models is also delicate. Aggressive fine-tuning can concentrate updates in a few experts, hurting general capability. The recipes that work involve carefully scoped adaptation and, in some cases, freezing the router entirely.
“MoE is what happens when you optimize for the constraint everyone forgot about: how much compute you can afford at inference time.”
What this means for builders
Most developers will never touch expert routing directly. What matters is that the price-performance frontier for hosted models is now defined by MoE variants. Expect the ratio of quality to per-token cost to keep improving faster than dense-only progress alone would predict.
For teams considering open-weight deployment, MoE models are more efficient per FLOP but require more VRAM to hold all experts. The trade-off can favor dense models on constrained hardware and MoE on well-provisioned clusters.
Key Topics
Extended Knowledge
- Switch Transformer (2021) and GLaM (2022) laid the modern MoE foundations that current frontier models build on.
- Expert-choice routing, introduced in 2022, addresses the load-balancing problems that plagued earlier top-k routing schemes.
- The gap between total and active parameters is the number to watch in modern model cards.
Frequently Asked
Not always. MoE models excel when total capacity matters and inference compute is constrained. Dense models can be simpler to fine-tune and deploy on limited hardware.
Look for a discrepancy between total and active parameter counts in the model card. Most current frontier models disclose this.
It affects it favorably at scale but introduces communication overhead. Latency depends more on serving infrastructure than architecture in most real deployments.
