M2: Distributed ML Algorithms on Arshad Siddiqui

Parallelisation of ML Algorithms

Mon, 01 Jan 0001 00:00:00 +0000

Parallelisation of ML Algorithms #

Machine-learning algorithms expose different kinds of parallel work. The correct decomposition depends on whether independent work occurs across records, features, trees, clusters, kernel entries, or optimisation steps.

Course coverage:

Problem decomposition for parallel machine learning
Ensemble methods and XGBoost-style tree ensembles
Parallel k-means assignment and centroid reduction
Distributed decision trees and random forests
Support vector machine parallelisation using block and MapReduce-style computation

Learning Objectives #

By the end of this page, you should be able to:

Communication-Aware Distributed ML

Mon, 01 Jan 0001 00:00:00 +0000

Communication-Aware Distributed ML #

Distributed machine learning is effective only when saved computation exceeds the cost of moving data and synchronising workers. Algorithm design must therefore account for message size, message frequency, barriers, and parameter placement.

Course coverage:

Communication overhead, illustrated through distributed k-means
Model parallelism when parameters do not fit or compute must be divided
Distributed k-nearest neighbours: partitioning, indexing, and approximate search
Gradient descent, SGD, and mini-batch optimisation
Synchronous versus asynchronous updates

Learning Objectives #

By the end of this page, you should be able to:

Clusters, Hadoop, and Spark

Mon, 01 Jan 0001 00:00:00 +0000

Clusters, Hadoop, and Spark #

Scale-out frameworks distribute data and computation across networked machines. Their value comes from aggregate compute and storage, while their main costs are communication, scheduling, serialisation, and repeated synchronisation.

Course coverage:

Cluster computing with Hadoop and Spark
MapReduce k-means, mini-batch and streaming variants
Convergence and communication reduction
Distributed convolutional neural networks: data and model parallelism
Layer or tensor slicing, mixed precision, pipelining, and distributed execution frameworks

Learning Objectives #

By the end of this page, you should be able to:

Distributed Training Strategies

Mon, 01 Jan 0001 00:00:00 +0000

Distributed Training Strategies #

Distributed training combines devices to increase throughput, fit larger models, or shorten time to solution. The central design question is what to partition: examples, model parameters, layers, tensors, or the training schedule.

Course coverage:

Data parallelism
Model parallelism
Pipeline parallelism and micro-batches
Gradient checkpointing
Mixed-precision training and loss scaling

Learning Objectives #

By the end of this page, you should be able to:

compare data, model, and pipeline parallelism
calculate global batch size, speedup, efficiency, and effective throughput
explain gradient aggregation through all-reduce
calculate simple pipeline time and utilisation
quantify the memory–computation trade-off of checkpointing
explain mixed precision and loss scaling

Big Picture #

flowchart TD
 A["Training Constraint"] --> B["More Data Throughput"]
 A --> C["Model Too Large"]
 A --> D["Activation Memory Too High"]
 B --> E["Data Parallelism"]
 C --> F["Model or Pipeline Parallelism"]
 D --> G["Checkpointing or Mixed Precision"]

 style A fill:#E1F5FE
 style B fill:#C8E6C9
 style C fill:#FFF9C4
 style D fill:#EDE7F6
 style E fill:#E1F5FE
 style F fill:#C8E6C9
 style G fill:#FFF9C4

1. Data Parallelism ☆ #

Each worker holds a complete model replica and processes a different shard of the mini-batch.