M1: Foundations

M1: Foundations #

This module establishes how to measure ML system performance, identify bottlenecks, and decide which work can run in parallel. It connects algorithmic complexity with the capabilities and limits of CPUs, GPUs, and clusters.

Module coverage:

• Performance Metrics and Complexity Analysis: time and space complexity, throughput, latency, and speedup through Amdahl’s Law.
• Parallelisation Paradigms: data-level, task-level, and algorithm-specific parallelism.
• Modern Hardware Architectures: multi-core CPUs, GPGPUs, GPU/CPU clusters, and resource management.

Learning Objectives #

• Distinguish complexity from measured running time, throughput, and latency.
• Explain the limits of speedup and the cost of communication.
• Choose between data, task, and request parallelism for a workload.
• Relate memory hierarchy, bandwidth, and hardware organisation to performance.

Pages #

Topic Outline #

1. ML and DL System Performance #

• Time complexity of algorithms and running time
• Performance scaling and tuning
• Training versus deployment
• Distributed and cloud environments, embedded devices, and mobile systems
• Throughput and response time

2. Parallel and Distributed Algorithms #

• Systems and performance
• Speedup: approaches, issues, and Amdahl’s Law ☆
• Data parallelism, task parallelism, and request parallelism
• Scale-out clusters: communication cost and its impact on speedup
• Scale-up versus scale-out

3. Parallel Programming Models #

• Parallel programming models and the MapReduce pattern
• Task-parallel and request-parallel execution
• Multi-core and GPGPU parallelism: SIMD versus MIMD
• Memory hierarchies and bandwidth considerations
• CPU-only and GPU-accelerated distributed clusters
• Horizontal and vertical data sharding
• Parameter-server and all-reduce paradigms

How This Module Connects #

Performance measurement and parallel execution provide the basis for M2: Distributed ML Algorithms. Communication and memory limits remain relevant throughout the later modules.

M2: Distributed ML Algorithms

M2: Distributed ML Algorithms #

This module applies parallel and distributed computing to ML algorithms. It examines how to divide computation, combine results, and balance communication against useful work during training.

Module coverage:

• Parallel ML Algorithms: CNNs, gradient descent and SGD, SVM, k-means, k-nearest neighbours, decision trees, and random forests.
• Distributed Training Strategies: data parallelism, model parallelism, pipeline parallelism, gradient checkpointing, and mixed-precision training.

Learning Objectives #

• Decompose ML algorithms into independent and dependent work.
• Explain how local results are aggregated and where communication is required.
• Compare distributed execution through Hadoop and Spark.
• Choose training strategies that address compute and memory constraints.

Pages #

Topic Outline #

4. Parallelisation of ML Algorithms #

• Problem decomposition and parallelisation
• Ensemble methods and XGBoost on CPUs
• k-means
• Distributed decision trees and random forests
• Distributed implementations using Spark ML and XGBoost
• Parallel SVM kernel computation using block-wise and MapReduce-style methods

5. Communication-Aware Distributed ML #

• Communication overhead in distributed algorithms
• Distributed k-means and model parallelism
• k-NN: data partitioning, locality-sensitive hashing, and distributed indexing
• KD-tree and Ball-tree indexing versus brute-force MapReduce
• Accuracy–latency trade-offs in approximate k-NN
• Gradient descent and SGD: mini-batch, synchronous, and asynchronous variants

6. Clusters, Hadoop, and Spark #

• Cluster computing with Hadoop and Spark
• Lloyd’s k-means algorithm in MapReduce
• Mini-batch k-means and streaming variants
• Convergence criteria and communication reduction through compressed updates
• CNN data and model parallelism, including layer splitting and tensor slicing
• Mixed precision, pipeline parallelism, and distributed training frameworks

7. Distributed Training Strategies ☆ #

• Data parallelism
• Model parallelism
• Pipeline parallelism
• Gradient checkpointing
• Mixed-precision training

How This Module Connects #

Use M1: Foundations to interpret complexity and speedup. Continue to M3: Scale-Out Systems for the platforms and communication mechanisms that support larger training runs.

M3: Scale-Out Systems

M3: Scale-Out Systems #

This module examines the platforms, communication patterns, and hardware organisation used to train across larger systems. It extends distributed algorithms into coordinated training services and federated settings.

Module coverage:

• ML Platforms and Frameworks: the Parameter Server model, Spark MLlib, and TensorFlow/PyTorch distributed training.
• Distributed Deep Learning: decentralised SGD, all-reduce, asynchronous parallelism, and large-scale neural networks.
• Federated Learning: training that preserves data privacy, communication efficiency, and hardware utilisation.

Learning Objectives #

• Explain how training platforms organise model state, workers, and updates.
• Compare centralised parameter services with collective communication.
• Assess the effects of bandwidth, latency, locality, and stale updates.
• Relate federated training to data distribution, privacy goals, and communication limits.

Pages and Planned Topics #

The outline below describes the full module scope. Planned topics have no chapter page yet. Chapter 9 currently covers parameter-server execution policies and communication optimisation; collective communication, matrix multiplication, and Local SGD remain to be added from their source material.

M4: Constrained Systems

M4: Constrained Systems #

This module considers how to make models practical when memory, compute, latency, and energy are limited. It connects model compression with edge deployment and the trade-offs required to retain useful prediction quality.

Module coverage:

• Model Optimisation Techniques: model compression, structured and unstructured pruning, post-training quantisation, and quantisation-aware training.
• Edge Deployment: TinyML, TensorFlow Lite, ONNX Runtime, and knowledge distillation.
• Performance Trade-offs: accuracy versus latency, model size versus throughput, and energy consumption optimisation.

Learning Objectives #

• Compare methods for reducing model size, computation, and memory use.
• Distinguish different quantisation and pruning approaches.
• Explain the role of knowledge distillation and deployment frameworks.
• Evaluate accuracy, latency, throughput, model size, and energy together.

Planned Topics #

This index records the module scope. Chapter pages can be added here as their source material becomes available.

ML Platforms and Frameworks

ML Platforms and Frameworks #

Training at scale requires more than dividing data between GPUs. A platform must decide where model state lives, who computes gradients, how updates are combined, and how information moves between devices.

This page focuses on the Parameter Server model and the practical constraints that determine whether adding resources makes training faster.

Topics covered:

• Parameter servers, workers, and independent scaling of storage and computation
• Training-state memory and parameter sharding
• Distributed loss, gradient aggregation, and the pull–compute–push cycle
• Communication costs, RDMA, MPI, PCIe, and NVLink
• Global batch size, learning-rate scaling, and warm-up
• Monitoring, fault recovery, and sparse versus dense workloads

Learning Objectives #

By the end of this page, you should be able to:

Distributed Deep Learning: Synchronisation and Communication

Distributed Deep Learning: Synchronisation and Communication #

Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide when to update the model, how much outdated information to accept, and how to reduce communication.

Relevant themes from the content outline:

• Asynchronous parallelism and parameter staleness
• Effects on convergence and variance
• System and program optimisation for neural networks
• Communication–computation trade-offs

The Parameter Server model provides the architectural foundation. Here, the focus is on the policies that determine its performance.

ML System Optimisation

ML System Optimisation #

ML System Optimisation studies how to make machine learning workloads faster, more scalable, more memory-efficient, and suitable for different hardware platforms.

The subject connects machine learning algorithms with the systems that train and deploy them: multi-core CPUs, GPUs, distributed clusters, cloud platforms, edge devices, and embedded systems.

ML system optimisation = model quality + computational efficiency + hardware awareness + scalability

The learning path begins with performance measurement and parallel computing, progresses through distributed machine learning and scale-out platforms, and concludes with model compression and resource-constrained deployment.