Fixing the mlx-vlm "batched vision path does not support KV cache quantization yet" error

You asked mlx-vlm to quantize the KV cache while running more than one image-and-text prompt at once. That combination isn't wired up yet. Here's what the error means and four ways to get generation running again.

The full message reads:

ValueError: the mlx-vlm batched vision path does not support kv cache quantization yet

It shows up at load or generation time, usually the moment you pass a kv_bits argument together with a batch of prompts through a vision-language model. Nothing is corrupt, and your model file is fine. mlx-vlm is refusing an unsupported combination up front instead of producing wrong results halfway through decode. That's the good version of a failure.

What the two halves of the error mean

Two independent features are colliding.

The batched vision path is the code that runs when you hand mlx-vlm more than one sequence at a time: several prompts, several images, or a prompt list with batch_size > 1. Batching multiple multimodal sequences means padding them to a common length and threading image embeddings through the language model together. It's a separate execution path from the single-sequence generate loop most examples use.

KV cache quantization is a decode-time memory optimization. During generation the model keeps a key/value cache for every past token; quantizing it to 8-bit or 4-bit (the kv_bits, kv_group_size, and quantized_kv_start knobs) shrinks that cache so long contexts fit in less memory. It's unrelated to how the model's weights are quantized.

The batched multimodal path in mlx-vlm simply hasn't implemented the quantized-cache branch yet. So when you request both at once, mlx-vlm raises rather than silently ignoring one of them. The word "yet" in the message is the tell: this is a missing feature, not a misconfiguration on your side.

Fix 1: drop KV-cache quantization on the batched path

The direct fix is to stop quantizing the cache while you're batching. Remove kv_bits (and the related kv_group_size / quantized_kv_start arguments) from the batched call:

# Triggers the error:
output = generate(model, processor, prompts, images, kv_bits=4, batch_size=8)

# Works: batched, full-precision KV cache
output = generate(model, processor, prompts, images, batch_size=8)

A full-precision KV cache costs more memory per sequence, but for the short-to-medium contexts typical of vision prompts, that's usually affordable. If you were quantizing the cache only to save a little headroom, this is the cheapest path forward.

Fix 2: run batch size 1

The single-sequence generate loop does support a quantized KV cache. If your reason for quantizing the cache is long context rather than throughput, you can keep kv_bits and drop batching:

for prompt, image in zip(prompts, images):
    out = generate(model, processor, prompt, image, kv_bits=4)

You trade batched throughput for a quantized cache. On Apple Silicon the decode step is memory-bandwidth-bound, so processing sequences one at a time is often less of a loss than it looks, and you get the cache savings you came for. Pick this when context length is the constraint; pick Fix 1 when throughput is.

Fix 3: quantize the weights, not the cache

A lot of people reach for kv_bits when what they actually want is a smaller model. Those are different quantizations. Weight quantization (converting the model to 4-bit with mlx_vlm.convert) is fully supported on the batched vision path and is where almost all the memory savings live. The KV cache is small next to the weights until context gets very long.

So convert the model to 4-bit weights and batch freely, with a full-precision cache:

# Quantize weights ahead of time; unrelated to KV-cache quantization
# python -m mlx_vlm.convert --hf-path <model> --q-bits 4 -q

output = generate(model, processor, prompts, images, batch_size=8)

One nuance worth carrying over from our multimodal work: the vision encoder and the language model don't have to be quantized the same way. In our LoRA fine-tuning of multimodal models we found it pays to treat the vision tower and the model.language_model.* submodules as separate targets rather than letting one rule blanket the whole graph. The same separation applies to quantization: the vision encoder is a small fraction of the parameters and among the most sensitive to precision loss, so leaving it in higher precision while quantizing the bulky language model is often the better quality-per-byte trade. MLX supports per-module quantization overrides for exactly this. It won't make the batched-cache error go away, but it's the right way to shrink a VLM without the cache-quantization detour.

Fix 4: update mlx-vlm

Because the limitation is scoped to "the batched vision path" and flagged as not supported "yet," it's the kind of gap that gets filled in a later release. Before you restructure your code, update:

pip install --upgrade mlx-vlm mlx

If a newer version has wired quantized KV cache into the batched path, your original call starts working unchanged. If it still raises, fall back to one of the first three fixes. Pin the version you land on so a future environment rebuild doesn't silently reintroduce the behavior.

Which fix to pick

• Batching for throughput and the cache was incidental: Fix 1, drop kv_bits.
• Quantizing the cache for long context: Fix 2, batch size 1, keep kv_bits.
• You actually wanted a smaller model: Fix 3, quantize weights, keep a full-precision cache, and consider leaving the vision encoder in higher precision.
• Any of the above: Fix 4, upgrade first in case the feature landed.

None of this is a workaround for a bug. It's picking a supported combination. MLX gives you weight quantization, KV-cache quantization, and batched multimodal generation as independent tools; the batched path just doesn't yet compose with a quantized cache.

Where this fits in our MLX work

We spend a lot of time at the edges of MLX, quantizing and serving large models on Apple Silicon, and the pattern behind this error is one we see constantly: a feature works on the common single-sequence path and hasn't reached the batched or multimodal path, or a dtype is fine in one place and overflows in another. Our writeup on MLX quantization pitfalls catalogs several of these, including a silent bfloat16 serialization bug and a GPU command-buffer timeout on large lazy-loaded models. If you're pushing MLX past the tutorials, the broader lesson in how we use a Mac as a compression lab is that unified memory buys you a lot of room, but the framework's newer optimizations don't always cover every code path yet. Read the error literally, and pick the combination that's actually implemented.