https://arshadhs.github.io/tags/gradient-compression/Recent content in Gradient Compression on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/090-distributed-deep-learning/<h1 id="distributed-deep-learning-synchronisation-and-communication"> Distributed Deep Learning: Synchronisation and Communication <a class="anchor" href="#distributed-deep-learning-synchronisation-and-communication">#</a> </h1> <p>Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide <strong>when to update the model, how much outdated information to accept, and how to reduce communication</strong>.</p> <p>Relevant themes from the content outline:</p> <ul> <li>Asynchronous parallelism and parameter staleness</li> <li>Effects on convergence and variance</li> <li>System and program optimisation for neural networks</li> <li>Communication–computation trade-offs</li> </ul> <!-- Outline alignment: _COURSE_MLSysOps_ZG516.pdf, page 6. This page covers the parameter-server material supplied for this topic. Collective communication, matrix multiplication, and Local SGD remain separate coverage. --> <p>The <a href="https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/">Parameter Server model</a> provides the architectural foundation. Here, the focus is on the policies that determine its performance.</p>