MLX quantized model memory footprint and tokens per second on an M4 Max

Two numbers decide whether a model runs well on your Mac: how many gigabytes it occupies, and how many tokens per second it decodes. Both come from arithmetic you can do on the back of an envelope, and quantization moves both in your favor at once.

People ask us the same question in two forms. "Will a 31B model fit on my M4 Max?" and "will it be fast enough to use?" These feel like separate problems. They're the same problem seen from two sides, because on Apple Silicon the thing that determines footprint (bytes of weights) is also the thing that determines decode speed (bytes of weights read per token). Get the first number right and the second follows.

The footprint is bits-per-weight times parameters

A model's weight memory is almost entirely one product:

bytes = parameters × bits_per_weight / 8

That's it. Everything else (KV cache, activations, framework overhead) is a correction on top. Run the numbers for a 31B-parameter dense model at a few precisions:

The real anchor: we compressed Gemma 4 31B with our RAM method to a 31 GB build, exactly half of its ~62 GB BF16 size, and it matched the published BF16 MMLU-Pro score. Half the size works out to about 8 bits per weight on average. The point of the table is that you can predict the footprint of any quantization target before you download anything: pick your bits-per-weight, multiply, done.

One honest correction to the arithmetic: "4-bit" is never exactly 4 bits per weight. Group-wise quantization stores a scale (and sometimes a zero-point) per group of weights, so the true cost is 4 bits plus the amortized scale. Smaller groups mean better quality and slightly more overhead. Our RAM research found that going from group size 128 to group size 32 adds roughly 0.125 bytes per parameter (about 1 extra bit) while noticeably improving accuracy. So a "4-bit" MLX model realistically lands near 4.5 bits per weight. Budget for that, not for a clean 4.

The KV cache is the part people forget

Weights are static. The KV cache grows with context length and sits on top of the weight footprint:

kv_bytes ≈ 2 × layers × kv_heads × head_dim × context_len × dtype_bytes

For short prompts it's negligible next to the weights. For long contexts it can add several gigabytes, which is exactly why MLX offers KV-cache quantization (8-bit or 4-bit) as a separate knob from weight quantization. If you're planning to run a 31 GB model at long context on a 36 GB machine, the cache is what pushes you over the edge, not the weights.

What actually fits on an M4 Max

M4 Max ships with 36 to 128 GB of unified memory. Usable budget is lower than the sticker number: the OS, the compositor, your other apps, and the KV cache all want a share. A practical rule is to plan for roughly 75-80% of installed memory being available to the model. That gives you something like:

• 36 GB M4 Max: a 31 GB weight footprint fits, but with little headroom for long-context KV cache. A 4-bit 31B (~17 GB) is the comfortable choice.
• 64 GB M4 Max: the 31 GB Gemma 4 build runs with room to spare for context and other work.
• 128 GB M4 Max: you're into 70B-class territory at 4-bit, or the 31B model at 8-bit with a large cache.

Because allocation is data-free and fast, you don't have to guess a preset and hope. You name the budget and let the allocator solve for it. That's the whole premise behind why a modest Mac can prepare models at all, which we pushed to its limit when we showed a 16 GB Mac Mini quantizing a 250 GB model by streaming one shard at a time and never holding the full model in memory. Preparation memory and inference memory are different budgets: you can build a model far larger than you can serve.

Tokens per second is a bandwidth problem, not a compute problem

Here's the part that surprises people coming from GPU-land. During autoregressive decode, generating one token requires reading essentially every weight the token depends on, once, from memory. The matrix multiplies are tiny (one token wide). So decode isn't limited by how many FLOPs the chip can do; it's limited by how fast it can pull weights out of memory. Decode is memory-bandwidth-bound.

That gives you a ceiling you can compute:

tokens_per_second  ≤  memory_bandwidth / bytes_read_per_token

An M4 Max moves roughly 400 to 550 GB/s depending on the exact die. Take the 31 GB Gemma build: dividing bandwidth by footprint puts the theoretical ceiling in the mid-teens of tokens per second. Drop the same model to 4-bit (~17 GB) and the ceiling roughly doubles, because you're reading half as many bytes per token. This is a roofline, not a benchmark. Real throughput sits below it, cache behavior, sampling, and prompt processing all take a cut, and the number you actually see depends on your model, your context length, and your MLX version. We're deliberately not quoting a measured tokens-per-second figure here, because the honest one is "it depends on your setup." What doesn't depend on your setup is the direction: fewer bytes per weight means a higher ceiling.

That's the second reason quantization matters. It's easy to think of quantization as purely a way to make a model fit. On Apple Silicon it's also a way to make it faster, because the same reduction in bytes that shrinks the footprint also shrinks the per-token memory traffic that caps decode speed.

Two things that bend the rule

Mixture-of-experts. An MoE model only activates a subset of its parameters per token, so decode bandwidth scales with active parameters, not total. A model that's huge on disk can decode like a much smaller one. Footprint and speed decouple.

Unified memory. There's no separate VRAM pool and no PCIe copy between "host" and "device" memory. The weights live in one address space the GPU reads directly. That's why the bandwidth number in the roofline is the system memory bandwidth, and why a Mac with enough unified memory can run models that would need multiple discrete GPUs.

How to plan a deployment

1. Fix your footprint first. Decide bits-per-weight from your quality bar, multiply by parameters, add a KV-cache allowance for your target context. That's your memory number.
2. Check it against usable memory, not installed memory. Reserve ~20-25% for the OS and overhead.
3. Estimate the speed ceiling as bandwidth divided by footprint, then expect to land below it. If the ceiling is already too low, a lower bit-width raises it, at some quality cost.
4. Prepare on any Mac, serve on the right one. Building a quantized model is a streaming, low-memory job; serving it needs the whole thing resident. Don't conflate the two budgets.

For teams that want this decided by measurement rather than arithmetic, with the capability audit that proves the compressed model still does its job, that's what Shepherd is for. But the arithmetic above is enough to know, before you download a single safetensors shard, whether a model will fit on your M4 Max and roughly how fast it will run.