https://arshadhs.github.io/tags/distributed-deep-learning/Recent content in Distributed Deep Learning on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/<h1 id="m3-scale-out-systems"> M3: Scale-Out Systems <a class="anchor" href="#m3-scale-out-systems">#</a> </h1> <p>This module examines the platforms, communication patterns, and hardware organisation used to train across larger systems. It extends distributed algorithms into coordinated training services and federated settings.</p> <p>Module coverage:</p> <ul> <li><strong>ML Platforms and Frameworks:</strong> the Parameter Server model, Spark MLlib, and TensorFlow/PyTorch distributed training.</li> <li><strong>Distributed Deep Learning:</strong> decentralised SGD, all-reduce, asynchronous parallelism, and large-scale neural networks.</li> <li><strong>Federated Learning:</strong> training that preserves data privacy, communication efficiency, and hardware utilisation.</li> </ul> <!-- Coverage follows _COURSE_MLSysOps_ZG516.pdf, Content Structure on page 3; topic boundaries follow the Learning Plan on pages 5–6. --> <h2 id="learning-objectives"> Learning Objectives <a class="anchor" href="#learning-objectives">#</a> </h2> <ul> <li>Explain how training platforms organise model state, workers, and updates.</li> <li>Compare centralised parameter services with collective communication.</li> <li>Assess the effects of bandwidth, latency, locality, and stale updates.</li> <li>Relate federated training to data distribution, privacy goals, and communication limits.</li> </ul> <h2 id="pages-and-planned-topics"> Pages and Planned Topics <a class="anchor" href="#pages-and-planned-topics">#</a> </h2> <table> <thead> <tr> <th style="text-align: right">Chapter</th> <th>Topic</th> <th>Availability</th> </tr> </thead> <tbody> <tr> <td style="text-align: right">8</td> <td><a href="https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/">ML Platforms and Frameworks</a></td> <td>Page available; currently focuses on the Parameter Server model</td> </tr> <tr> <td style="text-align: right">9</td> <td><a href="https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/">Distributed Deep Learning: Synchronisation and Communication</a></td> <td>Page available; parameter-server synchronisation, compression, overlap, and placement</td> </tr> <tr> <td style="text-align: right">10</td> <td>Locality and Large-Scale Parallelism</td> <td>Planned</td> </tr> <tr> <td style="text-align: right">11</td> <td>GPU Architecture and Federated Learning</td> <td>Planned</td> </tr> </tbody> </table> <p>The outline below describes the full module scope. Planned topics have no chapter page yet. Chapter 9 currently covers parameter-server execution policies and communication optimisation; collective communication, matrix multiplication, and Local SGD remain to be added from their source material.</p>https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/<h1 id="distributed-deep-learning-synchronisation-and-communication"> Distributed Deep Learning: Synchronisation and Communication <a class="anchor" href="#distributed-deep-learning-synchronisation-and-communication">#</a> </h1> <p>Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide <strong>when to update the model, how much outdated information to accept, and how to reduce communication</strong>.</p> <p>Relevant themes from the content outline:</p> <ul> <li>Asynchronous parallelism and parameter staleness</li> <li>Effects on convergence and variance</li> <li>System and program optimisation for neural networks</li> <li>Communication–computation trade-offs</li> </ul> <!-- Outline alignment: _COURSE_MLSysOps_ZG516.pdf, page 6. This page covers the parameter-server material supplied for this topic. Collective communication, matrix multiplication, and Local SGD remain separate coverage. --> <p>The <a href="https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/">Parameter Server model</a> provides the architectural foundation. Here, the focus is on the policies that determine its performance.</p>