https://arshadhs.github.io/tags/sgd/Recent content in SGD on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m2-distributed-ml-algorithms/050-communication-aware-distributed-ml/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m2-distributed-ml-algorithms/050-communication-aware-distributed-ml/<h1 id="communication-aware-distributed-ml"> Communication-Aware Distributed ML <a class="anchor" href="#communication-aware-distributed-ml">#</a> </h1> <p>Distributed machine learning is effective only when saved computation exceeds the cost of moving data and synchronising workers. Algorithm design must therefore account for message size, message frequency, barriers, and parameter placement.</p> <p>Course coverage:</p> <ol> <li><strong>Communication overhead</strong>, illustrated through distributed k-means</li> <li><strong>Model parallelism</strong> when parameters do not fit or compute must be divided</li> <li><strong>Distributed k-nearest neighbours:</strong> partitioning, indexing, and approximate search</li> <li><strong>Gradient descent, SGD, and mini-batch optimisation</strong></li> <li><strong>Synchronous versus asynchronous updates</strong></li> </ol> <h2 id="learning-objectives"> Learning Objectives <a class="anchor" href="#learning-objectives">#</a> </h2> <p>By the end of this page, you should be able to:</p>