Distributed Training Strategies

Distributed Training Strategies #

Distributed training combines devices to increase throughput, fit larger models, or shorten time to solution. The central design question is what to partition: examples, model parameters, layers, tensors, or the training schedule.

Course coverage:

1. Data parallelism
2. Model parallelism
3. Pipeline parallelism and micro-batches
4. Gradient checkpointing
5. Mixed-precision training and loss scaling

Learning Objectives #

By the end of this page, you should be able to:

• compare data, model, and pipeline parallelism
• calculate global batch size, speedup, efficiency, and effective throughput
• explain gradient aggregation through all-reduce
• calculate simple pipeline time and utilisation
• quantify the memory–computation trade-off of checkpointing
• explain mixed precision and loss scaling

Big Picture #

flowchart TD
    A["Training Constraint"] --> B["More Data Throughput"]
    A --> C["Model Too Large"]
    A --> D["Activation Memory Too High"]
    B --> E["Data Parallelism"]
    C --> F["Model or Pipeline Parallelism"]
    D --> G["Checkpointing or Mixed Precision"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Data Parallelism ☆ #

Each worker holds a complete model replica and processes a different shard of the mini-batch.

ML Platforms and Frameworks

ML Platforms and Frameworks #

Training at scale requires more than dividing data between GPUs. A platform must decide where model state lives, who computes gradients, how updates are combined, and how information moves between devices.

This page focuses on the Parameter Server model and the practical constraints that determine whether adding resources makes training faster.

Topics covered:

• Parameter servers, workers, and independent scaling of storage and computation
• Training-state memory and parameter sharding
• Distributed loss, gradient aggregation, and the pull–compute–push cycle
• Communication costs, RDMA, MPI, PCIe, and NVLink
• Global batch size, learning-rate scaling, and warm-up
• Monitoring, fault recovery, and sparse versus dense workloads

Learning Objectives #

By the end of this page, you should be able to:

Distributed Deep Learning: Synchronisation and Communication

Distributed Deep Learning: Synchronisation and Communication #

Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide when to update the model, how much outdated information to accept, and how to reduce communication.

Relevant themes from the content outline:

• Asynchronous parallelism and parameter staleness
• Effects on convergence and variance
• System and program optimisation for neural networks
• Communication–computation trade-offs

The Parameter Server model provides the architectural foundation. Here, the focus is on the policies that determine its performance.

ML System Optimisation

ML System Optimisation #

ML System Optimisation studies how to make machine learning workloads faster, more scalable, more memory-efficient, and suitable for different hardware platforms.

The subject connects machine learning algorithms with the systems that train and deploy them: multi-core CPUs, GPUs, distributed clusters, cloud platforms, edge devices, and embedded systems.

ML system optimisation = model quality + computational efficiency + hardware awareness + scalability

The learning path begins with performance measurement and parallel computing, progresses through distributed machine learning and scale-out platforms, and concludes with model compression and resource-constrained deployment.