https://arshadhs.github.io/tags/hadoop/Recent content in Hadoop on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m2-distributed-ml-algorithms/060-clusters-hadoop-and-spark/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m2-distributed-ml-algorithms/060-clusters-hadoop-and-spark/<h1 id="clusters-hadoop-and-spark"> Clusters, Hadoop, and Spark <a class="anchor" href="#clusters-hadoop-and-spark">#</a> </h1> <p>Scale-out frameworks distribute data and computation across networked machines. Their value comes from aggregate compute and storage, while their main costs are communication, scheduling, serialisation, and repeated synchronisation.</p> <p>Course coverage:</p> <ol> <li><strong>Cluster computing with Hadoop and Spark</strong></li> <li><strong>MapReduce k-means</strong>, mini-batch and streaming variants</li> <li><strong>Convergence and communication reduction</strong></li> <li><strong>Distributed convolutional neural networks:</strong> data and model parallelism</li> <li><strong>Layer or tensor slicing, mixed precision, pipelining, and distributed execution frameworks</strong></li> </ol> <h2 id="learning-objectives"> Learning Objectives <a class="anchor" href="#learning-objectives">#</a> </h2> <p>By the end of this page, you should be able to:</p>