Model Compression Foundations #
A model can be too large to download, too demanding to hold in memory, or too slow to run. Compression can address different parts of that problem. Start by identifying which resource is limiting deployment, then measure whether the compressed model solves that problem while retaining useful prediction quality.
- Parameter reduction, pruning, quantisation, and low-rank factorisation
- Weight sharing, clustering, and lossless coding
- Knowledge distillation, temperature, soft targets, and loss functions
- Storage, runtime memory, and hardware-aware latency
1. Decide What to Reduce #
| Quantity | Meaning | How to check |
|---|---|---|
| Stored size | Bytes needed to save or download the model | Measure the actual file, including metadata |
| Runtime memory | Resident weights, peak live activations, and workspace | Profile peak use on the device |
| Traffic | Bytes moved between memory levels, devices, or servers | Profile transfers for the target workload |
| Compute | Arithmetic needed for the chosen implementation | Count operations and measure latency |
For P dense weights and b bytes each, nominal storage is P × b. Parameter reduction changes P; quantisation changes how values are represented. Pruning can create zeros without changing the dimensions of a dense tensor; to shrink the file or skip work, the representation and execution must use those zeros.
Lossy compression changes model values or structure, as in pruning, clustering or reduced precision. Lossless encoding restores the encoded symbols exactly. A lossy transformation can be followed by a lossless encoding of its result.
GPU CNNs map convolutions to parallel kernels. Shared kernels across spatial positions are an architectural example of weight sharing. See Parallel and Distributed Convolution for how the underlying operations divide across workers.
2. Low-Rank Factorisation ☆ #
A dense layer can map a large input vector through a smaller intermediate vector if its weight matrix is well approximated by two thin matrices. For W with m rows and n columns, choose rank r and factors A of shape m × r and B of shape r × n. With column-vector input x, compute Bx first and then apply A.
\[ W\approx AB,\qquad Wx\approx A(Bx),\qquad P_{\mathrm{dense}}=mn,\qquad P_{\mathrm{factorised}}=r(m+n) \]A 4,096 × 4,096 layer with rank 256 falls from 16,777,216 weights to 2,097,152: 8× fewer weights. FP32 storage falls from 64 MiB to 8 MiB before metadata. Multiply–accumulates per input follow the same ideal ratio here, but two kernel launches and intermediate storage can affect latency. Factorisation only reduces the parameter count if r(m+n) is smaller than mn.
Truncated singular-value decomposition retains the r largest singular values and gives a best rank-r approximation in Frobenius norm. The discarded squared singular values sum to the squared approximation error. Choose a rank by validating task accuracy as well as matrix error; those are different measures.
3. Share Values and Encode the Result #
Weight sharing makes multiple connections refer to the same trainable value. It may be built into a model, as in a convolutional kernel, or imposed during compression. HashedNets uses a deterministic hash to map connections to shared buckets. Weight clustering groups weights by similarity, replaces each with its centroid, and stores a codebook with indices.
For N weights, K centroids and b bits per centroid, packed fixed-width indices give this nominal storage:
\[ S_{\mathrm{clustered}}=N\lceil\log_2 K\rceil+Kb\quad\text{bits} \]Example: Eight FP32 weights use 256 bits. Three FP32 centroids use 96 bits, and eight two-bit indices use 16 bits. The total is 112 bits and the compression ratio is 256/112 ≈ 2.29×, before format overhead. For a group containing 0.92, 0.88, and 0.95, the centroid is about 0.917. The index decodes exactly to the centroid, but the original weight is approximated.
Huffman coding assigns shorter codes to more frequent symbols. Arithmetic coding encodes a sequence using a probability-weighted interval. Both can preserve the encoded symbols exactly, but neither restores the original weights after clustering. Check whether a deployment runtime can use the compressed representation or must expand it first.
4. Knowledge Distillation ☆ #
A teacher produces a distribution over classes. A smaller student learns from true labels and from the teacher’s probabilities, which can reveal that some wrong classes are more plausible than others. The student is the model deployed afterwards.
At temperature T, softmax divides logits z by T. Raising T softens the distribution. Use the same temperature for teacher and student in the matching term.
\[ p_i^{(T)}=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}, \qquad L=\alpha\,\mathrm{CE}(y,p_s^{(1)}) +(1-\alpha)T^2\mathrm{KL}\!\left(p_t^{(T)}\,\|\,p_s^{(T)}\right) \]Here α weights the hard-label loss, CE is cross-entropy, and KL compares the teacher distribution with the student’s. The T² factor compensates for the temperature-dependent gradient scale in this formulation. Ordinary hard-label training uses T = 1 in its loss. The distillation temperature is a training hyperparameter.
flowchart TD
T["Teacher predictions"] --> S["Soft targets"]
S --> L["Combined training loss"]
H["True labels"] --> L
P["Student predictions"] --> L
L -->|"Update student"| P
style T fill:#EDE7F6
style S fill:#E1F5FE
style H fill:#FFF9C4
style P fill:#C8E6C9
style L fill:#E1F5FE
Worked Example: One Training Input #
Teacher logits are (5, 2, 1), student logits are (3, 2, 1.5), and the true class is 0. Use T = 4 and α = 0.5.
| Class | Teacher at T = 4 | Student at T = 4 |
|---|---|---|
| 0 | 0.5434 | 0.4055 |
| 1 | 0.2567 | 0.3158 |
| 2 | 0.1999 | 0.2787 |
At T = 1 the student assigns class 0 about 0.6285, giving hard loss −ln(0.6285) ≈ 0.4644. The teacher-to-student KL divergence is about 0.03945. The scaled soft loss is 16 × 0.03945 ≈ 0.6312, and the combined loss is about 0.5478. Use unrounded probabilities when reproducing the result.
5. Compression Ratios and Device Budgets ☆ #
\[ \mathrm{CR}=\frac{S_{\mathrm{original}}}{S_{\mathrm{compressed}}}, \qquad M_{\mathrm{peak}}\approx M_{\mathrm{resident\ weights}}+M_{\mathrm{peak\ live\ activations}}+M_{\mathrm{workspace}} \]A compression ratio of 4× means the result uses one quarter of the original bytes, a 75% reduction. Include codebooks, scales, indices and headers when computing an actual ratio. Peak runtime memory also depends on whether weights are loaded into RAM or read from separate storage.
Consider a device with an 8 MB shared inference budget, five million resident weights, and 2 MB of live activations. Use decimal MB and omit other workspace in this illustration.
| Representation | Weight size | Total with activations | Fits? |
|---|---|---|---|
| FP32 | 20 MB | 22 MB | No |
| FP16 | 10 MB | 12 MB | No |
| 8-bit | 5 MB | 7 MB | Yes |
| 8-bit after removing half of the weights structurally | 2.5 MB | 4.5 MB | Yes |
If activations rise to 4 MB, the 8-bit-only model now needs 9 MB and fails the budget. The structurally smaller model needs 6.5 MB under the same accounting. Measure actual peak memory; the example holds activation size fixed across weight formats.
6. Predict Which Change Can Improve Latency #
The roofline model gives an optimistic lower bound based on compute work F, peak compute rate Π, bytes moved B and memory bandwidth β:
\[ T\,\gtrsim\,\max\!\left(\frac{F}{\Pi},\frac{B}{\beta}\right) \]Suppose a layer needs 0.5 GFLOP at 1 TFLOP/s and reads 20 MB at 10 GB/s. Arithmetic takes at least 0.5 ms; moving the weights takes at least 2 ms. If 8-bit weights reduce traffic to 5 MB with unchanged arithmetic and no conversion cost, the two terms are both 0.5 ms. If structured pruning then halves both work and weight bytes, both terms fall to 0.25 ms. These are bounds, not benchmark results; kernel launches, activation traffic and unsupported low-precision operations can dominate.
Sparse execution also depends on hardware support. A supported 2:4 pattern stores two retained weights per group of four along the expected dimension plus position information; its benefit depends on datatype, layout and kernel. Irregular zeros alone do not imply fewer executed operations. See Pruning and Deep Compression for sparse storage calculations and Energy-Constrained ML for energy and data movement.
Quick Checks #
- Can a smaller file increase latency? Yes. Decoding, extra kernel launches, or unsupported sparse layouts can cost more time than the saved transfer.
- Why use distillation instead of only compressing an existing layer? It trains a separate smaller model to preserve useful prediction behaviour.
- What should you measure before deploying? Task quality, actual stored size, peak memory, and latency on the target hardware.
References and Further Reading #
- Han et al.: Deep Compression
- Chen et al.: Compressing Neural Networks with the Hashing Trick
- Hinton et al.: Distilling the Knowledge in a Neural Network
- Jaderberg et al.: low-rank expansions for CNNs
- Google AI Edge: model optimisation