Communication-Aware Distributed ML

Mon, 01 Jan 0001 00:00:00 +0000

Communication-Aware Distributed ML #

Distributed machine learning is effective only when saved computation exceeds the cost of moving data and synchronising workers. Algorithm design must therefore account for message size, message frequency, barriers, and parameter placement.

Course coverage:

Communication overhead, illustrated through distributed k-means
Model parallelism when parameters do not fit or compute must be divided
Distributed k-nearest neighbours: partitioning, indexing, and approximate search
Gradient descent, SGD, and mini-batch optimisation
Synchronous versus asynchronous updates

Learning Objectives #

By the end of this page, you should be able to:

ML Platforms and Frameworks

Mon, 01 Jan 0001 00:00:00 +0000

ML Platforms and Frameworks #

Training at scale requires more than dividing data between GPUs. A platform must decide where model state lives, who computes gradients, how updates are combined, and how information moves between devices.

This page focuses on the Parameter Server model and the practical constraints that determine whether adding resources makes training faster.

Topics covered:

Parameter servers, workers, and independent scaling of storage and computation
Training-state memory and parameter sharding
Distributed loss, gradient aggregation, and the pull–compute–push cycle
Communication costs, RDMA, MPI, PCIe, and NVLink
Global batch size, learning-rate scaling, and warm-up
Monitoring, fault recovery, and sparse versus dense workloads

Learning Objectives #

By the end of this page, you should be able to:

Communication on Arshad Siddiqui

Communication-Aware Distributed ML

Communication-Aware Distributed ML #

Learning Objectives #

ML Platforms and Frameworks

ML Platforms and Frameworks #

Learning Objectives #