https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/Recent content in M3: Scale-Out Systems on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/<h1 id="ml-platforms-and-frameworks"> ML Platforms and Frameworks <a class="anchor" href="#ml-platforms-and-frameworks">#</a> </h1> <p>Training at scale requires more than dividing data between GPUs. A platform must decide <strong>where model state lives, who computes gradients, how updates are combined, and how information moves between devices</strong>.</p> <p>This page focuses on the <strong>Parameter Server model</strong> and the practical constraints that determine whether adding resources makes training faster.</p> <p>Topics covered:</p> <ul> <li>Parameter servers, workers, and independent scaling of storage and computation</li> <li>Training-state memory and parameter sharding</li> <li>Distributed loss, gradient aggregation, and the pull–compute–push cycle</li> <li>Communication costs, RDMA, MPI, PCIe, and NVLink</li> <li>Global batch size, learning-rate scaling, and warm-up</li> <li>Monitoring, fault recovery, and sparse versus dense workloads</li> </ul> <h2 id="learning-objectives"> Learning Objectives <a class="anchor" href="#learning-objectives">#</a> </h2> <p>By the end of this page, you should be able to:</p>https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/<h1 id="distributed-deep-learning-synchronisation-and-communication"> Distributed Deep Learning: Synchronisation and Communication <a class="anchor" href="#distributed-deep-learning-synchronisation-and-communication">#</a> </h1> <p>Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide <strong>when to update the model, how much outdated information to accept, and how to reduce communication</strong>.</p> <p>Relevant themes from the content outline:</p> <ul> <li>Asynchronous parallelism and parameter staleness</li> <li>Effects on convergence and variance</li> <li>System and program optimisation for neural networks</li> <li>Communication–computation trade-offs</li> </ul> <!-- Outline alignment: _COURSE_MLSysOps_ZG516.pdf, page 6. This page covers the parameter-server material supplied for this topic. Collective communication, matrix multiplication, and Local SGD remain separate coverage. --> <p>The <a href="https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/">Parameter Server model</a> provides the architectural foundation. Here, the focus is on the policies that determine its performance.</p>