M2: Distributed ML Algorithms

Mon, 01 Jan 0001 00:00:00 +0000

M2: Distributed ML Algorithms #

This module applies parallel and distributed computing to ML algorithms. It examines how to divide computation, combine results, and balance communication against useful work during training.

Module coverage:

Parallel ML Algorithms: CNNs, gradient descent and SGD, SVM, k-means, k-nearest neighbours, decision trees, and random forests.
Distributed Training Strategies: data parallelism, model parallelism, pipeline parallelism, gradient checkpointing, and mixed-precision training.

Learning Objectives #

Decompose ML algorithms into independent and dependent work.
Explain how local results are aggregated and where communication is required.
Compare distributed execution through Hadoop and Spark.
Choose training strategies that address compute and memory constraints.

Pages #

Chapter	Page	Main focus
4	Parallelisation of ML Algorithms	Problem decomposition, ensembles, XGBoost, k-means, trees, forests, and SVM
5	Communication-Aware Distributed ML	Communication overhead, distributed k-means and k-NN, model parallelism, and SGD
6	Clusters, Hadoop, and Spark	Cluster frameworks, MapReduce k-means, distributed CNNs, and communication reduction
7	Distributed Training Strategies	Data, model, and pipeline parallelism, checkpointing, and mixed precision

Topic Outline #

4. Parallelisation of ML Algorithms #

Problem decomposition and parallelisation
Ensemble methods and XGBoost on CPUs
k-means
Distributed decision trees and random forests
Distributed implementations using Spark ML and XGBoost
Parallel SVM kernel computation using block-wise and MapReduce-style methods

5. Communication-Aware Distributed ML #

Communication overhead in distributed algorithms
Distributed k-means and model parallelism
k-NN: data partitioning, locality-sensitive hashing, and distributed indexing
KD-tree and Ball-tree indexing versus brute-force MapReduce
Accuracy–latency trade-offs in approximate k-NN
Gradient descent and SGD: mini-batch, synchronous, and asynchronous variants

6. Clusters, Hadoop, and Spark #

Cluster computing with Hadoop and Spark
Lloyd’s k-means algorithm in MapReduce
Mini-batch k-means and streaming variants
Convergence criteria and communication reduction through compressed updates
CNN data and model parallelism, including layer splitting and tensor slicing
Mixed precision, pipeline parallelism, and distributed training frameworks

7. Distributed Training Strategies ☆ #

Data parallelism
Model parallelism
Pipeline parallelism
Gradient checkpointing
Mixed-precision training

How This Module Connects #

Use M1: Foundations to interpret complexity and speedup. Continue to M3: Scale-Out Systems for the platforms and communication mechanisms that support larger training runs.

Distributed Training Strategies

Mon, 01 Jan 0001 00:00:00 +0000

Distributed Training Strategies #

Distributed training combines devices to increase throughput, fit larger models, or shorten time to solution. The central design question is what to partition: examples, model parameters, layers, tensors, or the training schedule.

Course coverage:

Data parallelism
Model parallelism
Pipeline parallelism and micro-batches
Gradient checkpointing
Mixed-precision training and loss scaling

Learning Objectives #

By the end of this page, you should be able to:

compare data, model, and pipeline parallelism
calculate global batch size, speedup, efficiency, and effective throughput
explain gradient aggregation through all-reduce
calculate simple pipeline time and utilisation
quantify the memory–computation trade-off of checkpointing
explain mixed precision and loss scaling

Big Picture #

flowchart TD
 A["Training Constraint"] --> B["More Data Throughput"]
 A --> C["Model Too Large"]
 A --> D["Activation Memory Too High"]
 B --> E["Data Parallelism"]
 C --> F["Model or Pipeline Parallelism"]
 D --> G["Checkpointing or Mixed Precision"]

 style A fill:#E1F5FE
 style B fill:#C8E6C9
 style C fill:#FFF9C4
 style D fill:#EDE7F6
 style E fill:#E1F5FE
 style F fill:#C8E6C9
 style G fill:#FFF9C4

1. Data Parallelism ☆ #

Each worker holds a complete model replica and processes a different shard of the mini-batch.

Distributed Training on Arshad Siddiqui

M2: Distributed ML Algorithms

M2: Distributed ML Algorithms #

Learning Objectives #

Pages #

Topic Outline #

4. Parallelisation of ML Algorithms #

5. Communication-Aware Distributed ML #

6. Clusters, Hadoop, and Spark #

7. Distributed Training Strategies ☆ #

How This Module Connects #

Distributed Training Strategies

Distributed Training Strategies #

Learning Objectives #

Big Picture #

1. Data Parallelism ☆ #