Black Sheep AI
Gemma 4 31B scores 85.2% on MMLU-Pro at full precision. The interesting question isn't that number, it's whether the number survives when you shrink the model to half its size. It does.
If you searched "gemma 4 mmlu pro," you're probably after one of two things: what the benchmark actually measures, or what happens to the score when the model gets quantized to run on hardware you own. This page answers both, then points you to the full run where we graded all 12,032 questions.
What MMLU-Pro measures
MMLU-Pro is the harder successor to the original MMLU. It's a multiple-choice academic exam spanning 14 subject categories, from math, physics, and chemistry through law, philosophy, and history. Two design choices make it a much stiffer test than the original:
- Ten answer options per question instead of four. On the old MMLU, random guessing got you 25%. On MMLU-Pro it gets you 10%. That change alone strips out a lot of the lucky-guess noise that used to inflate scores near the bottom of the range.
- Harder, more reasoning-heavy questions. Many items require multi-step work rather than fact recall, which is why weaker models tend to collapse on the STEM categories specifically.
The upshot: MMLU-Pro sorts genuinely capable models from ones that only look capable. A high score here is hard to fake.
What Gemma 4 scores
Google's published MMLU-Pro score for the full BF16 version of Gemma 4 31B is 85.2%. That's a strong result, comfortably in frontier territory for a model this size, and it's the number worth anchoring to when you compare anything derived from Gemma 4.
The score isn't uniform across subjects. Gemma 4 is strongest on the reasoning-heavy STEM categories and weakest on law, which has been a soft spot across the whole Gemma line at full precision, not a quirk of any one build. We break the per-category distribution down in the full results page; the headline to carry with you is 85.2% overall.
Why compression is the real question
Here's where most people's actual interest lies. A 31B model in BF16 weighs about 62 GB. That's workstation or multi-GPU territory, out of reach of most machines people actually own. Quantization shrinks it, but the worry is obvious: if you throw away precision in the weights, do you throw away the intelligence the benchmark is measuring?
This is where a benchmark like MMLU-Pro earns its keep. Perplexity, the metric most quantization work reports, is a proxy: it measures how well a model predicts held-out text, not whether it can answer a hard exam question. A model can post a respectable perplexity and still degrade on reasoning. Even worse, perplexity can mislead outright. We found cases where mean perplexity ranks the full-precision model as the worst of the bunch, the exact opposite of reality, because a handful of outlier sequences drag the average around. If you're judging a compressed model, the average of a proxy metric is not the thing to trust.
Downstream accuracy is. When we checked RAM-compressed models against ARC-Challenge and Winogrande, the reasoning benchmarks tracked the perplexity gains rather than contradicting them: the compression that preserved perplexity also preserved task accuracy. MMLU-Pro is the most demanding version of that same downstream test.
What happened to Gemma 4's score under compression
We compressed Gemma 4 31B to a 31 GB build using RAM, our data-free compression method, and ran the complete MMLU-Pro suite, all 12,032 questions, no subset, no early stopping. It scored 85.2%.
That's not "close to" the baseline or "graceful degradation." It's the same published number at half the size. Statistically the compressed model is indistinguishable from the full-precision one on this benchmark, and it now fits on a 48 GB unified-memory Mac instead of a multi-GPU rig. Same intelligence, on hardware people actually have.
The reason this works, briefly: uniform quantization spends the same number of bits on every weight, including the ones that matter most for reasoning. A mixed-precision method puts bits where they change the answer and pulls them from where they don't. The full methodology, the per-category breakdown, and the raw per-question results are on the canonical results page.
Quick answers
Is MMLU-Pro harder than MMLU? Yes. Ten answer options instead of four, and tougher questions. Random guessing scores 10%, not 25%.
What does Gemma 4 31B score on MMLU-Pro? 85.2% at full BF16 precision, per Google's published figure.
Does quantizing Gemma 4 lower its MMLU-Pro score? It doesn't have to. Our RAM-compressed 31 GB build matched the 85.2% baseline. Uniform low-bit quantization can lower it; a mixed-precision allocation that protects the sensitive weights doesn't.
Why not just trust perplexity? Because it's a proxy, and its average can rank models backwards. Downstream benchmarks like MMLU-Pro measure the capability you actually deploy.
Where can I see the full run? The complete 12,032-question results, per-category scores, and the downloadable raw answers are on the results page below.
If you want the methodology, the category-by-category numbers, and the raw grading data rather than the summary, read the full results.
