4-bit quantization for MoE models in Apple MLX

Quantizing a dense model to 4 bits is a solved, boring problem. Quantizing a mixture-of-experts model is not, because most of the weights are experts that barely fire, one small router decides everything, and MLX won't let you give two experts in the same layer different bit widths. Here's what actually changes, grounded in profiling 30,720 experts across two large MoE models.

Why MoE breaks the dense-model playbook

For a dense transformer, 4-bit quantization in MLX is close to a default. One weight matrix per projection, every parameter pulls its weight on every token, uniform --q-bits 4 at a sensible group size, done. The error is spread evenly because the work is spread evenly.

A mixture-of-experts model violates every part of that. Qwen3.5-397B carries 512 experts per layer across 60 layers, 30,720 expert instances in total, and routes each token to just 10 of them. Qwen3-235B has 128 experts per layer, 12,032 instances, top-8 routing. The overwhelming majority of those parameters sit idle for any given token. Treating all of them as equally important, which is what uniform 4-bit quantization does, spends your bit budget in the wrong places.

Three things make MoE quantization its own problem: the router is fragile, the experts are wildly unequal, and there's a long tail of experts that hardly do anything.

The router is a tiny weight matrix with veto power

Every MoE layer has a gate, a small router that scores the experts and picks the top-k. It's a rounding error's worth of parameters compared to the experts themselves, but if you damage it, every downstream expert computation is applied to the wrong tokens. Router precision matters far more per parameter than any single expert's does.

There's a subtler trap that our profiling surfaced: routing confidence doesn't mean the same thing across models. The softmax score the router assigns to its top pick scales inversely with how many experts are competing. Median top routing score was 0.078 in the 128-expert Qwen3-235B and 0.016 in the 512-expert Qwen3.5-397B. A routing decision that looks "confident" in a 512-expert model (say 0.048) would be below average in a 128-expert one. So any classification scheme that keys off routing scores, "this expert is critical if its score exceeds 0.1", has to be recalibrated per model. A fixed threshold that works for 128 experts classifies zero experts as critical at 512. We pulled the router apart layer by layer in our expert activation profiling, and the scale-dependence was the single most important finding for anyone trying to quantize these models by importance.

Experts are not equal, so bits shouldn't be either

Once you profile which experts actually fire, and how confidently, they sort into tiers. Running 150 calibration prompts through each model and capturing every activation, we classified all expert instances into critical, standard, deprioritized, and prunable:

When we allocated bits to match that importance, quality didn't just hold, it improved over uniform 4-bit. On Qwen3-235B, the profiling-driven allocation scored 76.7% on MMLU-Pro against 72.1% for a uniform 4-bit baseline, and it beat the BF16 reference (75.7%) too. HumanEval went from 78.7% at uniform 4-bit to 88.0%. The intuition holds up: spend 8 bits on the handful of experts that make the hard decisions, 2 bits on the ones that barely matter, and you come out ahead of spending 4 bits everywhere.

One counterintuitive result is worth flagging, because it's a trap. More high-precision experts is not monotonically better. We tried versions that pushed 8-bit allocation across 25, 35, then 40 layers; every one of them scored lower on MMLU-Pro than the version that used 8-bit on just 17 layers. Over-allocating precision to non-critical experts shifts the relative precision balance and hurts. The sweet spot is narrow.

The long tail, and why pruning is dangerous

The bigger the expert pool, the longer the tail of experts that almost never activate. In the 512-expert model, 18.1% of experts fired rarely enough to be flagged prunable, versus 6.6% in the 128-expert model. Four times the experts per layer leaves four times the room for redundancy. The distribution is layer-dependent too: early layers (0 through 18) were mostly redundant, with layer 0 alone showing 166 prunable experts, while the critical experts concentrated in the middle-to-late layers (25 through 45). Early MoE layers do broad, substitutable routing; later layers specialize.

That tail looks like free savings. It isn't, quite. Our profiling ran on 150 English-leaning calibration prompts, and an expert that only fires on Swahili grammar or Haskell monad questions won't register as important against a calibration set that never asks those questions. Domain specificity measured this way is deceptively low (mean 0.047 across domains), which flatters the case for pruning. When we later pruned on that basis, the blind spots bit back. The apparent redundancy of an expert pool is also a strong function of how many prompts you profile with, a point we dig into separately in what 100 prompts reveal about expert routing. Profile too little and you'll "discover" dead experts that are merely resting.

The MLX constraint that decides your options

Here's where the practical wall is. MLX's QuantizedSwitchLinear, the standard layer for quantized MoE experts, stores all of a layer's experts in one tensor with one shared element size. You cannot put expert 0 at 8-bit and expert 47 at 4-bit in the same layer. The framework gives you one bit width per layer, full stop.

So to get true per-expert bit widths we built MixedBitSwitchGLU: group the experts by bit width into separate QuantizedSwitchLinear instances, run every token through each group, and mask-and-combine so each token keeps only its routed group's output. It preserved quality beautifully, 15/15 collapse tests passed on the 397B model, and it saved 35.5 GB there. But running every token through every bit-group means doing the MoE math multiple times per layer. On Qwen3-235B that was about 30% overhead. On the 512-expert Qwen3.5-397B it was roughly 490%: about 8 seconds per prompt became about 47. That's unusable for anything interactive.

We abandoned it. For production we went back to layer-level uniform quantization with the standard MLX kernels, choosing one bit width per layer instead of per expert. That gave up the per-expert granularity but bought a 5.8x speedup, and the quality cost of layer-level versus expert-level decisions turned out to be negligible. The full story of the kernel, the quality wins, and why it still didn't ship is in our per-expert mixed-bit quantization writeup.

What to actually do with a MoE model on a Mac

1. Profile before you quantize. Which experts fire, how often, and how confidently is the input to every good decision here. Don't guess.
2. Protect the router. Keep gate weights at high precision. They're cheap and they're load-bearing.
3. Scale your thresholds by 1 / num_experts. Routing confidence dilutes with expert count; a threshold tuned on a 128-expert model will misclassify everything in a 512-expert one.
4. Be conservative about pruning the tail. Rare on your calibration set is not the same as unimportant. Broaden the calibration data before you delete anything.
5. Prefer layer-level bit allocation today. Until MLX ships native mixed-bit expert kernels, per-expert bit widths cost more in latency than they return in quality. Allocate at the layer, keep the standard kernels, ship something people will actually wait for.