https://arshadhs.github.io/tags/data-parallelism/Recent content in Data Parallelism on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/020-parallel-and-distributed-algorithms/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/020-parallel-and-distributed-algorithms/<h1 id="parallel-and-distributed-algorithms"> Parallel and Distributed Algorithms <a class="anchor" href="#parallel-and-distributed-algorithms">#</a> </h1> <p>Parallelisation divides computational work into parts that can execute concurrently. The purpose is to reduce completion time or increase throughput, but the gain depends on how much work is genuinely independent and how much overhead is introduced.</p> <p>This page covers:</p> <ul> <li>speedup, maximum speedup, and processor efficiency</li> <li>Amdahl&rsquo;s Law</li> <li>data-level parallelism</li> <li>task-level parallelism</li> <li>algorithm-specific parallelism</li> <li>communication, synchronisation, scheduling, and load-balancing overhead</li> <li>parallel merge sort and matrix multiplication</li> </ul> <h2 id="learning-objectives"> Learning Objectives <a class="anchor" href="#learning-objectives">#</a> </h2> <p>By the end of this page, you should be able to:</p>