Distributed Deep Learning

M3: Scale-Out Systems

AI, ML
ML System Optimisation, MLSysOps, Scale-Out Systems, Parameter Server, Distributed Deep Learning, Federated Learning, AI, ML

M3: Scale-Out Systems #

This module examines the platforms, communication patterns, and hardware organisation used to train across larger systems. It extends distributed algorithms into coordinated training services and federated settings.

Module coverage:

  • ML Platforms and Frameworks: the Parameter Server model, Spark MLlib, and TensorFlow/PyTorch distributed training.
  • Distributed Deep Learning: decentralised SGD, all-reduce, asynchronous parallelism, and large-scale neural networks.
  • Federated Learning: training that preserves data privacy, communication efficiency, and hardware utilisation.

Learning Objectives #

  • Explain how training platforms organise model state, workers, and updates.
  • Compare centralised parameter services with collective communication.
  • Assess the effects of bandwidth, latency, locality, and stale updates.
  • Relate federated training to data distribution, privacy goals, and communication limits.

Pages and Planned Topics #

ChapterTopicAvailability
8ML Platforms and FrameworksPage available; currently focuses on the Parameter Server model
9Distributed Deep Learning: Synchronisation and CommunicationPage available; parameter-server synchronisation, compression, overlap, and placement
10Locality and Large-Scale ParallelismPlanned
11GPU Architecture and Federated LearningPlanned

The outline below describes the full module scope. Planned topics have no chapter page yet. Chapter 9 currently covers parameter-server execution policies and communication optimisation; collective communication, matrix multiplication, and Local SGD remain to be added from their source material.

Distributed Deep Learning: Synchronisation and Communication

AI, ML
ML System Optimisation, MLSysOps, Distributed Deep Learning, Distributed SGD, Synchronisation, Gradient Compression, AI, ML

Distributed Deep Learning: Synchronisation and Communication #

Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide when to update the model, how much outdated information to accept, and how to reduce communication.

Relevant themes from the content outline:

  • Asynchronous parallelism and parameter staleness
  • Effects on convergence and variance
  • System and program optimisation for neural networks
  • Communication–computation trade-offs

The Parameter Server model provides the architectural foundation. Here, the focus is on the policies that determine its performance.