What “expert activated” means in a mixture-of-experts model

“Expert activated” isn’t a property of a model. It’s an event that happens per token, at every layer, thousands of times a second. Once you see it that way, a lot of MoE folklore about “dead” and “important” experts falls apart.

An expert activation is a per-token routing decision

A mixture-of-experts (MoE) layer replaces one big feed-forward network with many smaller ones called experts. In front of them sits a small network called the router, or gate. For every token that passes through the layer, the router does three things:

• Produces a logit for each expert (a raw score).
• Applies softmax across those logits to turn them into probabilities that sum to 1.
• Selects the top-k experts by probability and sends the token only to those.

That last step is the activation. When we say an expert was “activated,” we mean the router placed it in the top-k for a specific token at a specific layer. Nothing more. The other experts in that layer produced a probability too, they just lost the top-k cut for that token, so they sit idle and cost nothing.

This is why activation is a slippery word. It’s not “the model uses expert 214.” It’s “expert 214, at layer 37, was in the top-10 for this particular token.” Change the token and the set changes.

The volume is larger than most people expect. Take Qwen3.5-397B-A17B: 60 layers, 512 experts per layer, top-k of 10. A single token therefore triggers 10 × 60 = 600 expert activations on its way through the network. Qwen3-235B-A22B, with 94 layers and top-k 8, triggers 752. Multiply by every token in a prompt and you’re recording millions of events. When we profiled Qwen3.5-397B across 150 calibration prompts, we captured 7.8 million activation records in about 11 minutes.

The softmax score is a confidence, and it lies about magnitude

Each activation carries a number with it: the softmax probability the router assigned. It’s tempting to read that as “how strongly the router wanted this expert.” Roughly, yes. But the absolute value is almost meaningless on its own, because softmax normalizes across however many experts are competing.

Here’s the finding that caught us off guard. We profiled two models with the same tooling and compared their routing scores:

Softmax confidence scales inversely with expert count. A “very confident” routing decision in the 512-expert model, a P99 score of 0.048, would be a below-median score in the 128-expert model. Nothing is wrong with either model. With 512 experts splitting the probability mass, no single expert can claim much of it, even when it’s clearly the right choice.

The mechanism is just arithmetic. Softmax divides each exponentiated logit by the sum over all experts. Add more experts to the denominator and every individual probability shrinks, regardless of how peaked the underlying logits are. A router that is genuinely certain looks timid when it’s certain among 512 options instead of 128. This effect is consistent across our profiling of these architectures, from a 256-expert model up to the 512-expert Qwen3.5, and it’s covered in more depth in what 100 prompts reveal about expert routing.

Why this quietly breaks threshold rules

If you build any rule that reads the softmax score against a fixed number, that rule is secretly tied to the expert count of the model you tuned it on. A classifier like “score > 0.1 means this is a critical expert” picks out a sensible slice of a 128-expert model. Point it at the 512-expert model and it classifies zero experts as critical, because nothing clears 0.1. Same code, same threshold, completely different behavior, and no error to warn you.

The fix is to scale thresholds relative to 1/num_experts rather than hard-coding them. That value is the score every expert would get under perfectly uniform routing, so it’s the natural unit. A threshold expressed as “3× the uniform baseline” travels across architectures; a threshold of “0.1” does not.

This matters the moment you use activation data to make quantization decisions. We tier experts into precision bands, more bits for the experts that carry the most routed traffic, fewer for the rest, which requires classifying experts by confidence and frequency. Get the threshold scaling wrong and your whole tier assignment collapses onto one band. The mechanics of doing this per-expert without wrecking throughput are their own problem, which we work through in our writeup on per-expert mixed-bit quantization.

Activation frequency is a great signal, aimed at the wrong target

There’s a second trap hiding in the word “activated.” If an expert rarely lands in the top-k across your calibration set, it’s natural to call it redundant. It isn’t, and the size of your calibration set is doing more work than you think.

When we profiled a 256-expert model while varying how many prompts we fed it, the apparent redundancy moved enormously: at 5 prompts, roughly 30% of experts looked dead; at 100 prompts, only 0.6% did. That’s a 50× swing driven purely by how much of the input distribution you sampled. An expert that never fires on English prose may be the one carrying Spanish morphology or a specific code pattern. Small prompt sets don’t measure the model, they measure the narrowness of your prompts.

So the same activation-frequency signal supports two opposite actions, and only one is safe:

• Bit-width allocation (keep every expert, give the busy ones more bits): a defensible use of the signal.
• Pruning (delete the rare experts): the same signal, catastrophically misread.

Activation frequency tells you how often an expert is used. It does not tell you how much you’d miss it when it’s gone.

What to take away

• Treat “expert activated” as a per-token, per-layer event, not a static property. The interesting object is the distribution of activations, not any single one.
• Never compare raw softmax scores across models with different expert counts. Normalize by 1/num_experts first.
• Scale every threshold, for classification, tiering, or anything else, relative to the uniform baseline, so it survives a change in architecture.
• Use activation frequency to decide precision, not existence. It’s a strong compression signal and a dangerous deletion signal.
• Profile with enough prompt diversity that your rare experts get a fair chance to fire. What looks dead at 5 prompts is usually alive at 100.

The routing statistics we pulled from 30,720 expert instances, tier breakdowns, layer-level patterns, and the full profiling method, are in the research writeup.