ML Platforms and Frameworks

ML Platforms and Frameworks #

Training at scale requires more than dividing data between GPUs. A platform must decide where model state lives, who computes gradients, how updates are combined, and how information moves between devices.

This page focuses on the Parameter Server model and the practical constraints that determine whether adding resources makes training faster.

Topics covered:

• Parameter servers, workers, and independent scaling of storage and computation
• Training-state memory and parameter sharding
• Distributed loss, gradient aggregation, and the pull–compute–push cycle
• Communication costs, RDMA, MPI, PCIe, and NVLink
• Global batch size, learning-rate scaling, and warm-up
• Monitoring, fault recovery, and sparse versus dense workloads

Learning Objectives #

By the end of this page, you should be able to:

Distributed Deep Learning: Synchronisation and Communication

Distributed Deep Learning: Synchronisation and Communication #

Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide when to update the model, how much outdated information to accept, and how to reduce communication.

Relevant themes from the content outline:

• Asynchronous parallelism and parameter staleness
• Effects on convergence and variance
• System and program optimisation for neural networks
• Communication–computation trade-offs

The Parameter Server model provides the architectural foundation. Here, the focus is on the policies that determine its performance.