Distributed Deep Learning: Synchronisation and Communication #

Distributed training divides computation between workers, but those workers rarely finish together. A training system must decide when to update the model, how much outdated information to accept, and how to reduce communication.

Relevant themes from the content outline:

Asynchronous parallelism and parameter staleness
Effects on convergence and variance
System and program optimisation for neural networks
Communication–computation trade-offs

The Parameter Server model provides the architectural foundation. Here, the focus is on the policies that determine its performance.