https://arshadhs.github.io/tags/batch-size/Recent content in Batch Size on Arshad SiddiquiHugoen-ushttps://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/Mon, 01 Jan 0001 00:00:00 +0000https://arshadhs.github.io/docs/ai/038-ml-system-optimisation/m3-scale-out-systems/080-ml-platforms-and-frameworks/<h1 id="ml-platforms-and-frameworks"> ML Platforms and Frameworks <a class="anchor" href="#ml-platforms-and-frameworks">#</a> </h1> <p>Training at scale requires more than dividing data between GPUs. A platform must decide <strong>where model state lives, who computes gradients, how updates are combined, and how information moves between devices</strong>.</p> <p>This page focuses on the <strong>Parameter Server model</strong> and the practical constraints that determine whether adding resources makes training faster.</p> <p>Topics covered:</p> <ul> <li>Parameter servers, workers, and independent scaling of storage and computation</li> <li>Training-state memory and parameter sharding</li> <li>Distributed loss, gradient aggregation, and the pull–compute–push cycle</li> <li>Communication costs, RDMA, MPI, PCIe, and NVLink</li> <li>Global batch size, learning-rate scaling, and warm-up</li> <li>Monitoring, fault recovery, and sparse versus dense workloads</li> </ul> <h2 id="learning-objectives"> Learning Objectives <a class="anchor" href="#learning-objectives">#</a> </h2> <p>By the end of this page, you should be able to:</p>