Parallel and Distributed Algorithms
ML System Optimization,
MLSysOps,
Parallel Computing,
Amdahl's Law,
Data Parallelism,
Task Parallelism,
AI,
ML
Parallel and Distributed Algorithms #
Parallelisation divides computational work into parts that can execute concurrently. The purpose is to reduce completion time or increase throughput, but the gain depends on how much work is genuinely independent and how much overhead is introduced.
Course coverage:
- Systems and performance: execution time, throughput, efficiency, and bottlenecks
- Speedup—approaches and issues: Amdahl’s Law, serial work, overhead, and load imbalance
- Data parallelism versus task parallelism versus request parallelism
- Scale-out clusters: communication cost and its impact on speedup
The worked algorithms also show how divide-and-conquer and matrix operations expose parallel work.