Expert pruning in MoE models: pitfalls and practice

Three quantized variants scored 15 out of 15 on our automated tests. One of them couldn’t translate a sentence into Spanish without dropping a Chinese character into the middle of it. That gap between “passes the suite” and “works” is the whole story of expert pruning.

The failure that a green test suite hid

We pruned 18.1% of the experts from Qwen3.5-397B-A17B, 5,562 of its 30,720 expert instances, based on activation profiling. Every automated quality check we had said the model was fine. Then we read the outputs.

Here is the prompt: “Translate the following to Spanish: ‘The weather is beautiful today and I plan to go hiking in the mountains.’”

The unpruned model:

El clima está hermoso hoy y planeo ir de caminata a las montañas.

The pruned model:

El clima está hermoso hoy y我 plane to go hiking in the mountains.

It starts in Spanish, drops the Chinese character 我 (“I”) into the middle, then gives up and finishes in English. This passed our collapse test because the test checks for a minimum length of ten characters and has no keyword check for translation prompts. By the metric, it was a pass.

That wasn’t the only regression. Asked to explain what a monad is, the pruned model confidently denied that monads exist in functional programming. Given the fox-chicken-corn river-crossing puzzle, the unpruned model wrote a complete 1,119-character solution; the pruned model produced 206 characters that mostly restate the problem and never solve it. Both passed, because the tests only looked for a keyword and a length floor.

Why the tests lied

We ran the standard 15-prompt collapse suite against three variants. The results looked like a clean bill of health:

The pruned models had fewer warnings than the untouched baseline. Collapse tests check for minimum response length, keyword presence, repetition, and empty output. Those catch a model that has fallen over completely. They say nothing about a model that answers fluently and wrongly. Language contamination, factual denial, and truncated reasoning all sail straight through.

Activation frequency is not importance

The pruning itself is trivial and cheap. To remove an expert you don’t delete it; you set its row in the router’s gate weight to a large negative value so softmax drives its selection probability to zero:

# gate_weight shape: [num_experts, hidden_dim] = [512, 4096]
gate_weight[expert_idx, :] = -1e9

After softmax, that expert is never in the top-k. Zero runtime cost. The mechanism is sound. The target selection is where it goes wrong.

We picked experts to prune by activation frequency: anything that fired on less than 0.05% of tokens across 150 calibration prompts. The reasoning feels airtight, if an expert almost never activates, surely removing it costs almost nothing. It doesn’t hold, for a reason worth stating plainly: an expert that activates 0.03% of the time isn’t 30 times less important than one that activates 1% of the time. It might be the only expert that handles a rare but essential capability. A fire extinguisher is used 0% of the time during normal operations. That doesn’t make it safe to remove.

Our companion perplexity study makes the size of the danger concrete. On a 256-expert model, removing just the least-used 5% of experts, roughly 13 of 256 per layer, blew perplexity up by 13.2x, from 6.580 to 86.906. At 10% the model was effectively destroyed; at 25% it output noise. The full numbers are in why you can’t prune MoE experts, even the ones nobody uses. The router was trained with every expert present. Zero one out and the weighted sum it expects has a hole in it that no other expert was trained to fill.

The calibration set was the real culprit

The experts we pruned weren’t random. They were disproportionately the specialists our calibration set never exercised. All 150 prompts were primarily English, 25 per domain, and even the “multilingual” bucket held relatively simple translations. Against that input distribution, the experts carrying non-English generation, niche domain knowledge like functional programming, and extended multi-step reasoning barely fired. So they looked prunable. They weren’t; they were just off-camera.

This is the same sample-size trap we hit during activation profiling. Measured against 5 prompts, about 30% of a model’s experts can look dead. Measured against 100 diverse prompts, that figure falls to 0.6%. Small calibration sets don’t reveal redundancy, they manufacture the appearance of it. When you then prune on that appearance, you delete the model’s long tail of rare capabilities while keeping every common-case skill intact, which is exactly why the damage is invisible to English-centric smoke tests and obvious the moment someone asks for Spanish.

The benchmark breakdown pointed the same way. On MMLU-Pro the pruned model scored 43.6% overall, but History came in at 0% and Biology at 20%. Without a matched baseline run we can’t attribute every point of that to pruning, but a knowledge domain collapsing to zero is exactly the signature of removed knowledge-domain experts.

What to do instead

The lesson is not “never touch the expert pool.” It’s that activation frequency during calibration is a poor proxy for importance, and that quality gates built for collapse detection can’t see the failures pruning produces. If you’re going to prune anyway, do it like this.

• Prune only truly dead experts. Drop the threshold from “rarely fires” to “never fires.” Moving from < 0.05% to < 0.01% activation frequency shrinks the prune set for Qwen3.5-397B from 5,562 experts to a rough 800 to 1,200, with far higher confidence that nothing load-bearing goes with it.
• Fix the calibration set before you fix the model. Include every target language, real domain-specific content (formal logic, niche science, legal terminology), and weight the set toward the tail of expected usage rather than the head.
• Test past collapse. Add perplexity on diverse held-out text, domain-specific probes (multilingual translation, niche-knowledge questions), and manual spot-checks on adversarial prompts aimed at the capabilities you expect to be fragile.
• Prefer soft down-weighting to hard removal. Instead of setting a gate row to -1e9, multiply its magnitude by a small factor so the expert is rarely chosen but still available when a token genuinely needs it. That keeps the long-tail safety net in place.
• Prune in stages and measure at each one. Step through 2%, 5%, 10%, and 15%, checking quality at every stage. You’ll find the point where degradation starts before you’ve shipped past it.

There’s a better use for the same activation data: keep every expert and vary precision instead of existence. Tiered quantization gives the busy experts more bits and the quiet ones fewer, and in our measurements it costs a fraction of a percent of perplexity rather than orders of magnitude. Compress precision, not coverage.

The full pruning study, with the router-masking implementation, the complete failure transcripts, and the benchmark tables, is below.