Latent MoE
Latent MoE runs the routed feed-forward experts in a smaller latent space. It is still a sparse mixture-of-experts (MoE) layer. A router selects a few experts for each token, and only those experts process it. Nemotron 3, Kimi K3, and Muse Spark Open use this basic layout with different compression ratios and normalization details.
Basic idea
Run the selected experts in a narrower latent space
Compression
Nemotron 3: 4x
Kimi K3 and Muse Spark Open: 2x
Example architectures
Nemotron 3 Super 120B-A12B
Nemotron 3 Ultra 550B-A55B
Kimi K3
Muse Spark Open 816B-A42B
From regular MoE to Latent MoE
An MoE, or Mixture of Experts, is an ensemble model that combines several smaller “expert” subnetworks inside the GPT-like decoder architecture. A router assigns each token to a subset of these experts. The idea here is that by using multiple smaller subnetworks instead of one large network, MoEs aim to allocate computational resources more efficiently.
“Sparse” in the context of a “Sparse Mixture of Experts” refers to the fact that at any given time, only a subset of the expert layers are actively used for processing a token.
In a regular MoE layer, the routed experts operate directly at the model width. In Latent MoE, the routed path is first projected down into a smaller latent space, the experts operate there, and the result is projected back up.
Routing and the latent bottleneck affect different parts of the computation. Routing controls how many experts run. The bottleneck controls the width at which the selected experts run. The projection layers also cost compute, so a 4x narrower latent space does not imply a 4x speedup for the whole model.
In Muse Spark Open, the router reads the full 8,192-dimensional input and selects 8 of 256 experts. The expert path then projects down to 4,096 dimensions and applies RMSNorm. Each selected SwiGLU expert’s output is scaled and normalized before the weighted sum, which is projected back to 8,192 dimensions. There is no shared expert. The reference implementation shows the ordering of routing, projections, and normalization.
Scaling the bottleneck in Nemotron 3
The part I find most interesting is the Latent MoE idea introduced in Nemotron 3 Super.
For Super, this was 4096 -> 1024 -> 4096. For Ultra, it is 8192 -> 2048 -> 8192.
So the 4x compression ratio stays the same, but the model is scaled up substantially. Super has 120B total and 12B active parameters. Ultra grows to 550B total and 55B active parameters while retaining the same bottleneck ratio.
Kimi K3’s Stable LatentMoE
The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the Kimi K3 overview figure since it was already very crowded, but that’s essentially the same idea as in Nemotron 3 Ultra. The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
Kimi K3 uses a 7168 -> 3584 -> 7168 path. This halves the width, giving it a 2x bottleneck instead of Nemotron’s 4x bottleneck. Moonshot calls it Stable LatentMoE. It also applies RMSNorm before the up-projection, uses SiTU-GLU activations, and routes each token to 16 of 896 experts with Quantile Balancing.
Sources