question of gradients syncronization of resnet example

Question

ray provide an example for resnet distribution training. but the gradients syncronization is strange:

1. sync up weigths
2. train each worker a specific steps independently.
3. go back to step 1.

Is there any justification for this workflow?

I think it's neither sync or async method.

Answer 1

It's a slight generalization of normal synchronous SGD. If you train each worker for exactly one step in step 2, then it is regular batch SGD.