Distillation and quantization are treated as separate, sequential steps: first compress the knowledge, then compress the numbers. But what if a student model could be trained to absorb the teacher's knowledge and emerge with weight geometry that is already quantization-ready — in a single training run? SAKD proposes exactly that, building on SWAN's data-free sensitivity metrics and SAT's geometry regularisation to unify two historically independent compression stages into one.
The Problem: Two Compression Steps That Ignore Each Other
The standard pipeline for deploying a large language model at production scale involves two distinct compression phases. First, knowledge distillation: a compact student is trained to reproduce the teacher's representations, transferring capabilities across an architecture gap. Second, post-training quantization: the distilled student's weights are reduced from 16-bit to 4-bit or 8-bit precision for efficient inference. Each phase is a mature engineering discipline with its own literature, tooling, and failure modes.
The problem is that these two phases are almost always applied sequentially and independently. Distillation treats the student's weight geometry as an incidental output — something to be corrected after the fact if it causes quantization problems. Quantization treats the student as a fixed artifact to be compressed, with no influence over how that student was trained. The result is a pipeline with a fundamental blind spot: distillation can actively produce weight distributions that are hostile to quantization, and quantization has no mechanism to prevent this.
This blind spot is not theoretical. When a student is trained by feature-matching, it is optimised to reproduce teacher representations — including any outlier structure in those representations. If the teacher has layers with high-kurtosis weight distributions (as SWAN documents they frequently do), the student may develop correspondingly outlier-prone weights. These high-kurtosis distributions are precisely the ones that cause quantization damage: the extreme values dominate the quantization grid, compressing the range available for the bulk of well-behaved weights.
SAKD — SWAN-guided Knowledge Distillation — closes this blind spot. It applies the SWAN framework's data-free weight-geometry metrics to guide distillation supervision, and SAT's geometry regularisation to constrain student weight distributions during training. The result is a student that is simultaneously a better knowledge-transfer target and a better quantization candidate, without requiring a separate PTQ correction step.
The Three SAKD Mechanisms
SAKD introduces three complementary mechanisms, each addressing a different dimension of the distillation-quantization gap. Together they form a unified training objective that produces deployment-ready students in a single pass.
Sensitivity-Weighted Distillation Loss (SWDL)
Standard feature-matching distillation assigns equal loss weight to every teacher-student layer pair. SWDL replaces this uniform weighting with a principled scheme derived from the teacher's SWAN composite score — a weighted combination of four data-free metrics (excess kurtosis, SVD spectral concentration, output noise amplification, and reconstruction error proxy) computed directly on teacher weight tensors without any calibration data.
High-sensitivity teacher layers — those whose weight geometry indicates fragility under perturbation — receive proportionally stronger supervision. The weights are computed via a softmax with temperature annealing: training begins with broad, near-uniform supervision (useful when the student is far from the teacher everywhere) and progressively concentrates onto the high-sensitivity layers as the student masters the easier targets. Setting the temperature to infinity recovers standard uniform distillation, making SAKD a strict generalisation that can never perform worse than the baseline it replaces.
Student Geometry Regularisation (SGR)
SGR applies SAT's kurtosis ceiling and spectral conditioning directly to student weights during distillation training. The kurtosis regulariser uses a one-sided penalty: layers with excess kurtosis above a target ceiling (typically 1.5–2.5) are penalised, while layers with healthy distributions are left untouched. This preserves the natural expressiveness of well-behaved weight distributions while eliminating only the extreme outliers that cause quantization damage.
The spectral conditioning term minimises spectral concentration — the ratio of the largest singular value to the Frobenius norm — maintaining distributed singular-value spectra across student weight matrices. This serves a dual purpose: it makes weights more robust to the information loss inherent in quantization, and it bounds the spectral norm of weight updates, improving gradient stability during distillation training. The result is a student whose weight geometry is jointly optimised for knowledge transfer and quantization readiness. Without SGR, standard distillation optimises only the former and may actively harm the latter.
Targeted Distillation Noise Injection (TDNI)
TDNI injects calibrated quantization noise into student layers that are aligned to high-SWAN-score teacher layers — specifically, those in the top 20% of sensitivity. During each forward pass, the student's weight tensor is perturbed by uniform noise calibrated to the target deployment bit-width (typically 4-bit), and gradients pass through via the Straight-Through Estimator.
The effect is to simultaneously train two objectives in a single forward pass: the distillation loss trains the student to approximate the teacher's representations, while the noise injection trains the student's parameters to be robust to the quantization noise that will be applied at deployment. This is substantially more efficient than the standard pipeline of distillation followed by separate quantization-aware training (QAT). Crucially, TDNI applies noise only to high-sensitivity layers, avoiding disruption of stable layers whose training does not benefit from hardening — concentrating the effect where SWAN indicates it is most needed.
Why This Changes the Pipeline
The standard LLM compression pipeline has three discrete stages, each with its own overhead:
The key enabler is SWAN's data-free profiling. Because SWAN's four metrics are computed directly on weight tensors — excess kurtosis, SVD spectral concentration, output noise amplification, and reconstruction error proxy — the teacher profiling phase requires no calibration samples whatsoever. There is no need for task-specific data, no domain dependency, and no forward pass through the teacher with real inputs. The entire SWAN analysis completes in under 13 minutes on commodity hardware, even for models with 400 billion or more parameters. This makes the profiling overhead negligible relative to the distillation training itself.
After SAKD-distillation, the student's own SWAN scores serve directly as input to standard SWAN-guided PTQ if further quantization is desired — eliminating one full PTQ analysis step from the deployment pipeline. But the core promise is that many students will not need that additional step at all: their weight geometry, shaped by SGR and hardened by TDNI during training, will already be quantization-compatible.
Connection to the Research Trilogy
SAKD is the third entry in a research trajectory that uses weight-geometry sensitivity analysis as the organising principle for LLM optimisation. Each paper builds on the last, and together they offer a unified framework where weight geometry is a first-class concern at every stage of a model's lifecycle:
The pattern is clear: a single analytical framework — measuring how weight geometry predicts a layer's fragility under perturbation — applies across quantization, training, and distillation. The sensitivity metrics are the same. The insight is the same. The application changes, but the principle is consistent: not all layers are created equal, and acting on that inequality is the key to efficient model engineering at every stage.
This convergence is not accidental. SWAN establishes that weight-geometry metrics like kurtosis (ρ = 0.80 with quantization error) and output noise amplification (ρ = 0.69) are strong, non-redundant predictors of a tensor's sensitivity. SAT shows those metrics can be controlled during training. SAKD asks the natural next question: if we can measure teacher sensitivity data-free and control student geometry during training, why are these two capabilities not used together during distillation?
What This Means for Practitioners
Eliminate the PTQ step: students come out quantization-ready
The core practical promise of SAKD is collapsing a three-stage pipeline (train, distill, quantize) into two stages. SGR constrains student weight geometry during training, and TDNI co-trains quantization robustness in the same forward pass. If the hypotheses hold, the distilled student's weights will already have low kurtosis and distributed singular-value spectra — precisely the properties that enable clean quantization without post-hoc correction.
Data-free teacher profiling: no calibration data needed
SWAN's metrics are computed directly on weight tensors. There is no forward pass through the teacher with real inputs, no domain-specific calibration set, and no data pipeline to maintain. The entire teacher profiling phase takes less than 13 minutes for models up to 400B+ parameters. This is particularly valuable for teams that have access to a teacher's weights but limited proprietary training data — the sensitivity map is available immediately.
Framework-agnostic with explicit optimiser interaction analysis
SAKD works with any standard optimiser. The paper explicitly analyses the interaction between its geometry regularisers and AdamW's second-moment normalisation, identifying a potential feedback loop where kurtosis suppression reduces gradient magnitudes for outlier weights, causing AdamW to increase their adaptive learning rate. The recommended mitigation — monitoring kurtosis evolution and scaling regularisation coefficients — is described in detail. For teams using stateless optimisers like SGD or Muon, SWAN-style per-layer gradient normalisation can be applied directly.
Novel evaluation metric: SWAN Post-Distillation Audit
SAKD introduces a new way to evaluate distilled models: run SWAN's full four-metric analysis on the trained student. This produces a per-tensor sensitivity profile that directly predicts how well the student will quantize. The metric does not exist in prior distillation benchmarks and reframes evaluation as encompassing not only task performance but also deployment geometry — a student with low SWAN scores is a student ready for production.
Read the Full Paper
The complete SAKD paper includes the formal framework derivation, detailed algorithm specification, theoretical motivation for why SWAN scores approximate layer-wise Jacobian norms, the full proposed experimental protocol across four teacher-student pairs, hyperparameter reference tables, and a comprehensive discussion of limitations and open questions.
SAKD: SWAN-Guided Knowledge Distillation — Full Paper
huggingface.co/spaces/baa-ai/sakd-knowledge-distillationLicensed under CC BY-NC-ND 4.0 · Algorithmic Proposal
Need deep AI expertise to get your models into production?
Black Sheep AI helps organisations compress and distil large language models for efficient deployment — from sensitivity-aware distillation pipelines to quantization-ready architectures. Deep expertise, measurable quality improvements.
Talk to Our Team© 2026 baa.ai. All rights reserved.