Parameter Server

M3: Scale-Out Systems

AI, ML
ML System Optimisation, MLSysOps, Scale-Out Systems, Parameter Server, Distributed Deep Learning, Federated Learning, AI, ML

M3: Scale-Out Systems #

This module examines the platforms, communication patterns, and hardware organisation used to train across larger systems. It extends distributed algorithms into coordinated training services and federated settings.

Module coverage:

  • ML Platforms and Frameworks: the Parameter Server model, Spark MLlib, and TensorFlow/PyTorch distributed training.
  • Distributed Deep Learning: decentralised SGD, all-reduce, asynchronous parallelism, and large-scale neural networks.
  • Federated Learning: training that preserves data privacy, communication efficiency, and hardware utilisation.

Learning Objectives #

  • Explain how training platforms organise model state, workers, and updates.
  • Compare centralised parameter services with collective communication.
  • Assess the effects of bandwidth, latency, locality, and stale updates.
  • Relate federated training to data distribution, privacy goals, and communication limits.

Pages and Planned Topics #

ChapterTopicAvailability
8ML Platforms and FrameworksPage available; currently focuses on the Parameter Server model
9Distributed Deep Learning: Synchronisation and CommunicationPage available; parameter-server synchronisation, compression, overlap, and placement
10Locality and Large-Scale ParallelismPlanned
11GPU Architecture and Federated LearningPlanned

The outline below describes the full module scope. Planned topics have no chapter page yet. Chapter 9 currently covers parameter-server execution policies and communication optimisation; collective communication, matrix multiplication, and Local SGD remain to be added from their source material.

ML Platforms and Frameworks

AI, ML
ML System Optimisation, MLSysOps, Parameter Server, Distributed SGD, Communication, Batch Size, AI, ML

ML Platforms and Frameworks #

Training at scale requires more than dividing data between GPUs. A platform must decide where model state lives, who computes gradients, how updates are combined, and how information moves between devices.

This page focuses on the Parameter Server model and the practical constraints that determine whether adding resources makes training faster.

Topics covered:

  • Parameter servers, workers, and independent scaling of storage and computation
  • Training-state memory and parameter sharding
  • Distributed loss, gradient aggregation, and the pull–compute–push cycle
  • Communication costs, RDMA, MPI, PCIe, and NVLink
  • Global batch size, learning-rate scaling, and warm-up
  • Monitoring, fault recovery, and sparse versus dense workloads

Learning Objectives #

By the end of this page, you should be able to: