M3: Scale-Out Systems

Mon, 01 Jan 0001 00:00:00 +0000

M3: Scale-Out Systems #

This module examines the platforms, communication patterns, and hardware organisation used to train across larger systems. It extends distributed algorithms into coordinated training services and federated settings.

Module coverage:

ML Platforms and Frameworks: the Parameter Server model, Spark MLlib, and TensorFlow/PyTorch distributed training.
Distributed Deep Learning: decentralised SGD, all-reduce, asynchronous parallelism, and large-scale neural networks.
Federated Learning: training that preserves data privacy, communication efficiency, and hardware utilisation.

Learning Objectives #

Explain how training platforms organise model state, workers, and updates.
Compare centralised parameter services with collective communication.
Assess the effects of bandwidth, latency, locality, and stale updates.
Relate federated training to data distribution, privacy goals, and communication limits.

Pages and Planned Topics #

Chapter	Topic	Availability
8	ML Platforms and Frameworks	Page available; currently focuses on the Parameter Server model
9	Distributed Deep Learning: Synchronisation and Communication	Page available; parameter-server synchronisation, compression, overlap, and placement
10	Locality and Large-Scale Parallelism	Planned
11	GPU Architecture and Federated Learning	Planned

The outline below describes the full module scope. Planned topics have no chapter page yet. Chapter 9 currently covers parameter-server execution policies and communication optimisation; collective communication, matrix multiplication, and Local SGD remain to be added from their source material.

Federated Learning on Arshad Siddiqui

M3: Scale-Out Systems

M3: Scale-Out Systems #

Learning Objectives #

Pages and Planned Topics #