GPU

Parallel Programming Models

AI, ML
ML System Optimization, MLSysOps, Multi-Core CPU, GPGPU, SIMD, MIMD, SIMT, GPU, TPU, AI, ML

Parallel Programming Models #

Parallel algorithms need hardware that can execute independent work efficiently. Modern systems therefore combine multiple CPU cores, memory hierarchies, threads, instruction pipelines, GPUs, clusters, and specialised matrix processors.

Course coverage:

  1. Parallel programming models for expressing independent work
  2. MapReduce, task-parallel, and request-parallel patterns
  3. Multi-core CPUs and GPGPUs: SIMD, MIMD, SIMT, memory hierarchy, and bandwidth
  4. Distributed clusters: CPU-only and GPU-accelerated systems, data sharding, parameter servers, and all-reduce

The hardware sections also cover specialised matrix processors and the performance limits introduced by memory and communication.