Parallel Programming Models

ML System Optimization, MLSysOps, Multi-Core CPU, GPGPU, SIMD, MIMD, SIMT, GPU, TPU, AI, ML

Parallel Programming Models #

Parallel algorithms need hardware that can execute independent work efficiently. Modern systems therefore combine multiple CPU cores, memory hierarchies, threads, instruction pipelines, GPUs, clusters, and specialised matrix processors.

This page covers:

multi-core CPU organisation
cache and memory hierarchy
processes, threads, scheduling, and synchronisation
instruction pipelining and clock-cycle time
SIMD, MIMD, and SIMT execution
GPGPU architecture and GPU memory behaviour
CPU-only and GPU-accelerated clusters
Tensor Processing Units and systolic arrays

Learning Objectives #

By the end of this page, you should be able to: