Pruning and Deep Compression

Pruning and Deep Compression #

Pruning removes selected parts of a network, while deep compression combines multiple ways to store it more compactly. This page covers:

  • GPU CNN execution and federated learning
  • Structured and unstructured pruning
  • Pruning schedules and evaluation metrics
  • Compression pipelines and sparse storage

1. Execution Context #

  • GPU implementation of CNNs: uses parallel kernels for convolution and other operations. Irregular sparsity helps execution only if the kernels and hardware can exploit it.
  • Federated learning: trains across participants holding local data; smaller models or updates can reduce communication, but compression may affect convergence and participant compatibility.

2. Pruning Methods ☆ #

  • Unstructured magnitude-based pruning: zeros individual weights with small absolute values. It produces irregular sparsity without necessarily changing tensor dimensions.
  • Structured filter/channel pruning: removes whole filters, channels, or other groups, potentially producing smaller dense operations that ordinary hardware can execute efficiently.
  • One-shot pruning: applies the chosen pruning operation in one stage, usually followed by fine-tuning to recover quality.
  • Iterative pruning: alternates smaller pruning steps with retraining or fine-tuning, giving the model opportunities to adapt gradually.

3. Evaluation Metrics #

  • Sparsity ratio: the fraction of weights that are zero. For example, 800 zero weights out of 1,000 means 80% sparsity.
  • FLOPs reduction: the decrease in estimated floating-point operations. State whether the estimate counts dense operations or assumes sparse operations are skipped.
  • Accuracy drop: the reduction in prediction accuracy after compression; a change from 90% to 88% is a drop of two percentage points.

Parameter count and FLOPs do not determine actual runtime alone. Measure latency, memory, and model quality on the intended hardware.

4. Deep Compression and Sparse Storage #

  • Combining pruning, quantisation, and encoding: pruning removes connections, quantisation reduces distinct values or precision, and entropy encoding compresses the resulting representation.
  • Fine-tuning: updates the retained parameters after a transformation, aiming to recover quality lost during compression.
  • Sparse matrix formats: store nonzero values with location information instead of every zero. COO records coordinates; CSR stores values, column indices, and row boundaries.
  • Storage overhead: sparse indices consume space, so sparse storage is advantageous only when the saved zeros outweigh its metadata and format costs.
flowchart TD
    A["Trained network"] --> B["Prune and fine-tune"]
    B --> C["Quantise or share weights"]
    C --> D["Encode for storage"]
    D --> E["Check size and quality"]
    style A fill:#E1F5FE
    style B fill:#FFF9C4
    style C fill:#C8E6C9
    style D fill:#EDE7F6
    style E fill:#E1F5FE

This illustrates the classic compression sequence; deployment still needs a supported execution representation.

5. Calculate Sparse Storage Before Choosing a Format ☆ #

Magnitude pruning can be expressed as a binary mask. At threshold τ, keep entries whose magnitude is at least τ. If nnz entries remain in an m × n matrix, sparsity is:

\[ M_{ij}=\mathbf{1}\{|W_{ij}|\ge\tau\},\qquad W_{\mathrm{sparse}}=W\odot M,\qquad s=1-\frac{\mathrm{nnz}}{mn} \]

CSR (compressed sparse row) stores the nonzero values in row order, one column index per value, and m + 1 row pointers. Row i occupies entries from row_ptr[i] up to, but excluding, row_ptr[i+1]. The last pointer equals nnz; two equal consecutive pointers mean an empty row.

For b-value bytes per stored value, b-index bytes per column, and b-pointer bytes per row pointer:

\[ S_{\mathrm{CSR}}=\mathrm{nnz}(b_{\mathrm{value}}+b_{\mathrm{index}}) +(m+1)b_{\mathrm{pointer}},\qquad S_{\mathrm{dense}}=mn\,b_{\mathrm{value}} \]

Worked Example: 56.25% Sparse, Yet Larger Than Dense #

After pruning a 4 × 4 FP32 layer at threshold 0.1, seven of 16 weights remain. The values are (0.82, 0.31, 0.65, −0.44, 0.91, 0.57, −0.73), column indices are (0, 2, 1, 3, 2, 0, 3), and row pointers are (0, 2, 4, 5, 7). For example, the third row has entries in the half-open range [4, 5): only value 0.91 at column 2.

RepresentationCalculationBytes
Dense FP3216 × 464
CSR with four-byte indices and pointers7 × (4 + 4) + 5 × 476
CSR with one-byte indices and pointers, if supported7 × (4 + 1) + 5 × 140

These counts omit file headers and alignment. For a 1,000 × 1,000 layer at 90% sparsity with four-byte values, indices and pointers, CSR uses 804,004 bytes against 4,000,000 bytes dense, about 4.98× smaller before metadata. Small matrices pay more for row pointers relative to useful values. An index type must hold every stored column number and row offset.

COO can be convenient when assembling entries because each value carries its row and column. A bitmap uses one position bit per possible entry plus the retained values. The most compact format is not necessarily the fastest execution format; benchmark a supported kernel.

6. Connect the Pattern to the Hardware #

Structured pruning can remove entire output channels, allowing smaller dense kernels if dependent layers are adjusted. Some sparse kernels instead exploit a fixed pattern, such as keeping two weights in each group of four. Unstructured pruning leaves the original tensor shape and generally needs a sparse kernel to skip arithmetic. See Model Compression Foundations for the latency calculation.

References and Further Reading #


Home | M4: Constrained Systems