MLX quantization on Apple Silicon: a practical guide

The official MLX docs will get you a quantized model. They won't warn you that saving it back can silently corrupt every tensor, or that the wrong dtype turns a 397B model into a punctuation generator. Here's the working how-to, plus the traps we hit quantizing 400B+ parameter models on a Mac Studio.

What mlx_lm.convert actually does

Quantizing a local LLM with MLX is mostly one command. mlx_lm.convert downloads a Hugging Face model, replaces its linear layers with quantized equivalents, and writes an MLX-format model you can serve immediately:

mlx_lm.convert --hf-path Qwen/Qwen3-8B -q --q-bits 4 --q-group-size 64 --mlx-path ./qwen3-8b-4bit

The flags that matter:

• -q / --quantize turns quantization on. Without it you just get a format conversion.
• --q-bits sets the bit width. MLX's affine quantization supports 2, 3, 4, 6, and 8 bits. Default is 4.
• --q-group-size sets how many weights share one scale and bias. Default is 64. Smaller groups (32) capture the weight distribution more finely at a small storage cost; larger groups (128) save space.
• --dtype sets the precision of the unquantized parts. This one is a footgun. More below.

The Python API is the same underneath:

from mlx_lm import convert

convert(
    hf_path="Qwen/Qwen3-8B",
    mlx_path="./qwen3-8b-4bit",
    quantize=True,
    q_bits=4,
    q_group_size=64,
    dtype="bfloat16",
)

MLX uses group-wise round-to-nearest (RTN) quantization. There's no calibration data, no forward pass, no GPU cluster. On Apple Silicon's unified memory, the model you produce is the model you serve, which is why a Mac Studio can be a full compression workstation. We've written separately about how RAM turns a Mac into a model compression lab if you want the hardware-economics side of this.

Bit width and group size: the two knobs

Treat these as one joint decision, not two independent ones. Bit width is the headline number. Group size is the knob most people leave at the default and shouldn't.

A 4-bit model at group size 64 stores, per weight, four bits plus a shared scale and bias amortised across 64 values. Drop to group size 32 and you double the scale/bias overhead, about 0.125 extra bytes per parameter, but you get four times finer granularity. For many tensors that finer granularity buys more quality than spending the same bits on a higher bit width would. Apple Silicon supports group size 32 natively, so there's no kernel penalty for using it.

For a first pass, 4-bit at group size 64 is a sane default for an 8B dense model. When you go bigger, or lower than 4-bit, the defaults stop being safe, and the docs stop being helpful.

What the docs leave out

The MLX documentation covers the happy path. Everything below is what bit us on real, large models during our ExpertQuant work, and none of it is documented anywhere. Each one cost hours.

Pick your dtype based on the attention type, not habit

--dtype float16 looks like a reasonable choice, and for standard transformer attention (Qwen3, Llama) it's fine. For models with recurrent or state-space attention it's a silent killer.

Qwen3.5-397B uses GatedDeltaNet attention, which carries a recurrent state that accumulates across the sequence. Those accumulated values can blow past the float16 ceiling of plus/minus 65,504, overflow to NaN, and collapse the model. bfloat16 shares float32's exponent range, so it holds the accumulation. We converted the exact same model both ways: with float16 it produced garbage, with bfloat16 it passed all 15 of our collapse tests.

The rule we now follow:

• Standard attention (Qwen3, Llama, Mistral): float16 is fine.
• GatedDeltaNet, Mamba, RWKV, any recurrent or state-space attention: use --dtype bfloat16.

When in doubt, bfloat16 is the safer default across the board.

mx.save_safetensors can corrupt bfloat16 on the way out

This is the one that cost us the most. If your workflow loads a shard, edits a few tensors, and writes it back, do not round-trip bfloat16 through mx.save_safetensors (we saw this on mlx 0.29.3). The saved file has valid shapes and dtype metadata, opens without error, and contains numerically wrong bytes for every tensor in the shard, not just the ones you touched. No warning, complete model collapse at inference.

The fix is to do the load-modify-save with Hugging Face's safetensors library instead of MLX:

from safetensors import safe_open
from safetensors.torch import save_file

tensors = {}
with safe_open(shard_file, framework="pt") as f:
    for key in f.keys():
        tensors[key] = f.get_tensor(key)

# ... modify tensors ...

save_file(tensors, shard_file)

If you're not sure whether you've been bitten, hash each shard before and after your round-trip. A changed hash on a shard you only partially edited is the tell.

Very large models time out the GPU on first inference

mlx_lm.server lazy-loads weights on the first request, not at startup. For a small model that's fine. For a 237 GB model spread across a hundred-plus safetensors files, the first forward pass fires while the system is still paging weights into unified memory, and Metal's command-buffer timeout kills it:

[METAL] Command buffer execution failed:
Caused GPU Timeout Error (kIOGPUCommandBufferCallbackErrorTimeout)

In our testing on a 512 GB M3 Ultra, models up to roughly 200 GB served fine, and models above roughly 230 GB tripped the timeout on the first request. The workaround is to load eagerly with mlx_lm.load, which pulls everything in up front (about 38 seconds for that model) so the first real inference runs immediately.

A workflow that survives contact with large models

1. Convert with --dtype bfloat16 unless you have a specific reason to use float16, especially for anything with non-standard attention.
2. Run a quick inference immediately after conversion. Ask it "what is 2+2". A model that answers with repeated punctuation has collapsed, and you want to know that in ten seconds, not after an hour of evaluation.
3. If you post-process weights, use the safetensors library for the load-modify-save, never mx.save_safetensors on bfloat16.
4. For anything above ~200 GB, load with mlx_lm.load rather than serving cold.
5. Only then run your real evals.

That "verify before you evaluate" habit is the cheapest insurance in the whole pipeline. Quantization on Apple Silicon fails loudly when it fails, so a ten-second smoke test catches nearly everything.

If you're taking compressed models past your own experiments and into a production or fleet setting, capability verification stops being optional. That's the gap Shepherd is built to close: it takes RAM-compressed models from research to deployment with a capability audit attached, so "it quantized without crashing" isn't mistaken for "it's fine in production."