An architecture that replaces dense feed-forward layers with many "expert" sub-networks plus a router that activates only a few per token — Mixtral 8x7B, DeepSeek-V3, and widely believed to underpin GPT-4-class and other frontier models.
How it works
Mixture-of-Experts (MoE) architecture operates by replacing traditional dense feed-forward layers with multiple specialized sub-networks, known as experts. For each input token, a gating network evaluates all available experts and selects the top-k, typically just a couple out of a larger set, like 2 from 8. This design allows the model to maintain a high total parameter count—like the Mixtral 8x7B with around 47 billion parameters—while only activating a small subset of those parameters for any given token. This means that the computational load scales with the number of active experts rather than the total number of parameters, making it efficient in practice.
As a result, even though Mixtral has a vast number of parameters, its effective computational cost resembles that of a much smaller model, such as one with around 13 billion parameters. This efficiency is crucial for performance because it allows the model to handle complex tasks without incurring the full computational cost associated with all its parameters. However, there is a trade-off: all experts must reside in VRAM simultaneously, which means that while only a few are active during inference, the memory footprint still reflects the total number of parameters. This becomes a practical constraint for self-hosting, as the memory requirements can be significant.
To illustrate, imagine processing a sentence where the gating network identifies that only two out of eight experts are relevant for the task at hand. The model then activates these two experts to process the input, while the others remain dormant. This selective activation allows for efficient computation, ensuring that the model can deliver high-quality outputs without the overhead of running all experts at once. However, if the memory capacity is exceeded, it can lead to performance issues or even prevent the model from running altogether.
- Input token flows to gating network.
- Gating network selects top-k experts.
- Active experts process the token.
- Output is aggregated from active experts.
The software-engineering analogue
Think of MoE as a system where tasks are distributed among specialized workers, with a load balancer that routes each request to the appropriate worker. This is similar to how MoE routes tokens to specific experts based on learned criteria. However, the key difference lies in the routing mechanism: in MoE, the selection of experts is learned dynamically for each token, making it highly adaptive. Unlike traditional systems where only active workers are in memory, all experts in MoE must be resident in memory at all times, regardless of their current activity level. This means you can't simply scale down to only the busy workers; the entire set of experts is required for the routing process.
Why not the obvious alternative
A newcomer might first consider using a dense model that contains all the necessary knowledge without the complexity of a gating mechanism. However, this approach would lead to significantly higher FLOPs per token during inference. A dense model runs all parameters for every input, which means it incurs a heavy computational cost regardless of the actual relevance of the knowledge to the task at hand. This inefficiency becomes even more pronounced as the model size increases.
In contrast, MoE effectively decouples the model's capacity from its inference compute. By activating only a small number of experts, it allows for greater capability without a proportional increase in computational demands. This means that as frontier models grow in size, they can maintain affordability in serving costs, which is why many organizations are moving towards MoE architectures instead of sticking with dense models.
When to use which
Choosing between MoE and dense models largely depends on your operational constraints and goals. If you have sufficient VRAM to accommodate the total parameters of an MoE model, it can provide exceptional quality per active FLOP, making it a strong choice for high-performance applications. However, if you are working with memory constraints, such as on edge devices, a smaller dense model may be more appropriate. Dense models can be simpler to deploy and manage in environments where memory is limited, even if they lack the efficiency of MoE in terms of capacity versus compute.
In production: when it breaks
Why it happens. The router in the MoE architecture is not distributing tasks evenly among the experts, causing certain experts to be overloaded while others are underutilized. This load imbalance can stem from the router's learned preferences or from certain input characteristics that favor specific experts.
- Log token distribution across experts to identify use rates.
- Measure the performance impact of adding an auxiliary load-balancing loss to the training objective.
- Experiment with setting a capacity factor for each expert to limit the number of tokens they can handle.
What it delivers
MoE architecture delivers frontier-level capabilities while maintaining sub-linear inference compute costs. This efficiency is why large models can remain affordable to serve, enabling organizations to use advanced AI without incurring prohibitive operational expenses.