Sharding big models in MLX: the no_gpu_multi(split) path

A 16 GB Mac Mini can read, quantize, and write a 250 GB model without ever holding it in memory. That surprises people, so it tends to attract half-remembered API names. Let’s separate what MLX actually gives you from the search terms that get attached to it.

First, the honest note about “no_gpu_multi(split)”

If you searched for no_gpu_multi(split) expecting an MLX function with that signature, you won’t find one. It isn’t a symbol in MLX’s public API, and it isn’t a call in our conversion pipeline. We’re flagging that up front because the fastest way to lose a day is to grep for a function that was never real.

What the phrase gestures at is a genuine thing: placing and moving model shards through the compute path yourself instead of loading the whole model onto the device at once. On Apple Silicon that concept splits cleanly into two separate mechanisms, and conflating them is where most of the confusion lives:

• Shard streaming during conversion and quantization. This works, it’s cheap, and it’s how you fit a trillion-parameter checkpoint through a laptop.
• Splitting a model across the GPU for inference at sizes larger than memory. This does not have a free lunch on a single Mac, for a reason rooted in how unified memory and Metal work.

The rest of this guide is about telling those two apart, because the same word “sharding” means opposite things in each.

Why unified memory changes the question

On a discrete-GPU box, you have host RAM and separate VRAM, and “sharding” usually means chopping a model across several GPUs. Apple Silicon has neither of those axes. CPU and GPU share one pool of unified memory. There is no second card to split onto, and there is no host-to-device copy to amortize. So the interesting split isn’t spatial (across devices) but temporal: how much of the model is resident at any one instant.

That reframing is the whole trick behind fitting large models on small Macs. You never need the entire model in memory simultaneously. You need whatever slice you’re operating on right now, plus room to write the result.

The path that works: streaming conversion

The default tool, mlx_lm.convert(), materializes the entire BF16 model in unified memory before re-quantizing. For a dense 70B model that’s fine on a 192 GB Mac Studio. For a trillion-parameter MoE like Kimi-K2.6 or DeepSeek-V4, the BF16 intermediate would need on the order of 2 TB of RAM, which no Mac has.

A shard-streaming converter sidesteps that entirely. Source checkpoints already arrive as a sequence of model-XXXXX-of-YYYYY.safetensors files, typically 4 to 8 GB each. The converter reads the safetensors index to learn which tensor lives in which shard, then loops over output shards, and for each tensor it reads from the source shard, applies the bit-width from the allocation manifest, quantizes, and writes it out, closing both source and output shards before moving on. Peak memory holds at one source shard plus one output shard plus working space, roughly 15 GB total. The model size never enters the calculation.

We’ve run this end to end on Kimi-K2.6 (1.0 T parameters), DeepSeek-V4 (685 B), GLM-5 (450 B), and Llama-4-Maverick (400 B plus 16 experts), all at a peak below 15 GB. On a 250 GB source model, we’ve measured under 15 GB peak resident set at every stage on a 16 GB Mac Mini. The full trillion-parameter walkthrough has the loop and the memory profile.

Two things worth knowing before you reach for it:

• The output is byte-identical to what mlx_lm.convert() would produce given the same manifest and bit-widths. It’s a drop-in for the conversion step, not a different quantization method.
• Routed experts are handled at a uniform bit-width. In models with hundreds of experts per layer, per-expert measurement at conversion time is impractical, so the converter takes a single --expert-bits value (default 4) and applies it across all experts.

The cost is about 10% slower conversion than mlx_lm.convert(), from per-shard open and close overhead. On a 250 GB model that’s roughly three extra minutes. The real ceiling is disk, not memory.

The bottleneck moves to disk, and that’s a feature

Once memory stops being the constraint, disk becomes it. End-to-end conversion of a 250 GB BF16 model needs about 250 GB for the source (can live on an external SSD), ~25 GB for the quantized output, and ~10 GB of scratch, roughly 285 GB total. A 1 TB Mac Mini has room to spare. We’ve run the same pipeline off a Thunderbolt external SSD on a base 16 GB / 256 GB MacBook Air; sequential read tops out around 2.5 GB/s, so a full pass over 250 GB adds around 100 seconds of pure I/O on top of compute. The engineering rule that makes all of this hold is boring and strict: never materialize the full model. Iterate f.keys() with safetensors.safe_open, use mlx_lm.convert() rather than load-then-quantize-then-save, and the 250-GB-needs-256-GB-of-RAM assumption simply evaporates. The 16 GB Mac Mini writeup documents each stage’s measured peak.

The path that doesn’t have a free lunch

Here’s the honest boundary. Streaming solves preparation: converting, quantizing, verifying, building a release artifact. It does not solve inference of a model larger than your unified memory. During a forward pass the weights have to be addressable to the GPU, and streaming them in per layer means paying disk latency inside your token loop, which is not a viable interactive path. A 192 GB Mac Studio exists precisely for running the large model you prepared on the small one. Full-precision training and fine-tuning hit the same wall for the same reason: gradients and activations need the whole model resident.

And there’s a second, sharper wall on the training side that has nothing to do with bytes. When we trained LoRA adapters on the 128-expert MoE layers of Qwen3.5-35B-A3B, training died with:

[metal::malloc] Resource limit (499000) exceeded

That is not out-of-memory. A 192 GB Mac Pro hit it with only ~60 GB in use. Metal caps the number of buffer descriptors, distinct buffer handles, that can exist at once, and the ceiling is a hard 499,000. LoRA on a SwitchLinear layer spawns parameters for every expert, and the descriptor count scales roughly as num_layers × num_experts × rank × (forward + backward intermediates). At 128 experts across a few target layers, only rank 2 stays under the cap for a full run. Adding RAM changes nothing, because the limit is descriptor count, not bytes. The Metal buffer-limit investigation has the full rank-survival matrix and the workarounds we tried, most of which made it worse.

What to actually do

• If your source model exceeds about 60% of your unified memory, use a shard-streaming converter. Below that, plain mlx_lm.convert() is better tested and slightly faster.
• Provision disk, not RAM, for conversion work: source plus output plus roughly 10 GB scratch. An external Thunderbolt SSD is fine; you’ll pay I/O time, not correctness.
• Don’t expect streaming to let you run inference on a model bigger than memory. Prepare on the small Mac, serve on the big one.
• If you’re fine-tuning LoRA on a high-expert-count MoE through MLX, plan around rank 2 for the expert layers and don’t buy more RAM hoping to lift the rank. The wall is the Metal descriptor cap, not memory.
• Ignore any API name you can’t find in the MLX source or the pipeline code, including no_gpu_multi(split). The real levers are shard streaming, uniform expert bits, and disciplined per-tensor iteration.

The trillion-parameter conversion loop, with the exact memory profile and the shard-sizing defaults, is in the research writeup below.