Distributed Deep Learning: Synchronisation and Communication
ML System Optimisation,
MLSysOps,
Distributed Deep Learning,
Distributed SGD,
Synchronisation,
Gradient Compression,
AI,
ML
Distributed Deep Learning: Synchronisation and Communication #
Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide when to update the model, how much outdated information to accept, and how to reduce communication.
Relevant themes from the content outline:
- Asynchronous parallelism and parameter staleness
- Effects on convergence and variance
- System and program optimisation for neural networks
- Communication–computation trade-offs
The Parameter Server model provides the architectural foundation. Here, the focus is on the policies that determine its performance.