Mixture of experts: the feed-forward block, experts and the router

The feed-forward block, then DeepSeekMoE: 256 small experts plus a shared one, and a router that sends each token to its best eight.

This is the text of a chapter of the animated talk A Deep Seek into LLM Architecture by Ruben Galvão. Open the page with scripts on to play the slides, one key press per idea.

The feed-forward block (SwiGLU)

Zoom out: all this is the attention block, half a layer; the other half is the feed-forward block
Per token, no mixing: two wide matrices (7,168 → 18,432), one through SiLU (a gate), multiplied cell by cell, a third back to 7,168; adds to the stream
Attention mixes tokens; feed-forward thinks about each one
V3.2: first 3 layers dense, the rest experts

The feed-forward works per token

Darken all but the new token: feed-forward never looks at other tokens
Each column independent through the same three matrices
Position-wise feed-forward (original paper)
Nothing to cache; a new token costs one column

One expert: a narrower block

Mixture of experts: box the three matrices = an expert; make it narrower
Hidden 18,432 → 2,048 (V3.2 expert width): 9× smaller
Layer ≈ 584 M → ≈ 232 M weights
One thin expert isn’t a model: make many

256 experts

Shrink to one of 256 identical experts, plus one shared expert every token uses
Layer weights jump to ≈ 11.5 B
Sparsely-gated experts: Shazeer 2017; many small + shared: DeepSeekMoE
All 257 running: ≈ 26 B ops per token

The router: the top 8

Router: one 256 × 7,168 matrix scores the experts; top 8 + shared run, 248 skipped
8 results added, weighted by score (V3: sigmoid, normalised)
Layer holds ≈ 11.5 B weights; a token uses ≈ 0.59 B
Every layer has its own router; tokens pick different experts
Back down to under 4 B ops

Papers and sources

Noam Shazeer (2020): GLU Variants Improve Transformer (the gated feed-forward block)
Dai et al. (2024): DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models