M3: Scale-Out Systems
ML System Optimisation,
MLSysOps,
Scale-Out Systems,
Parameter Server,
Distributed Deep Learning,
Federated Learning,
AI,
ML
M3: Scale-Out Systems #
This module examines the platforms, communication patterns, and hardware organisation used to train across larger systems. It extends distributed algorithms into coordinated training services and federated settings.
Module coverage:
- ML Platforms and Frameworks: the Parameter Server model, Spark MLlib, and TensorFlow/PyTorch distributed training.
- Distributed Deep Learning: decentralised SGD, all-reduce, asynchronous parallelism, and large-scale neural networks.
- Federated Learning: training that preserves data privacy, communication efficiency, and hardware utilisation.
Learning Objectives #
- Explain how training platforms organise model state, workers, and updates.
- Compare centralised parameter services with collective communication.
- Assess the effects of bandwidth, latency, locality, and stale updates.
- Relate federated training to data distribution, privacy goals, and communication limits.
Pages and Planned Topics #
| Chapter | Topic | Availability |
|---|---|---|
| 8 | ML Platforms and Frameworks | Page available; currently focuses on the Parameter Server model |
| 9 | Distributed Deep Learning: Synchronisation and Communication | Page available; parameter-server synchronisation, compression, overlap, and placement |
| 10 | Locality and Large-Scale Parallelism | Planned |
| 11 | GPU Architecture and Federated Learning | Planned |
The outline below describes the full module scope. Planned topics have no chapter page yet. Chapter 9 currently covers parameter-server execution policies and communication optimisation; collective communication, matrix multiplication, and Local SGD remain to be added from their source material.