Mastering TensorFlow 1.x

Define and train the graph for synchronous updates

As discussed before, and depicted in the diagram here, in synchronous updates, the tasks send their updates to the parameter server(s), and ps tasks wait for all the updates to be received, aggregate them, and then update the parameters. The worker tasks wait for the updates before proceeding to the next iteration of computing parameter updates:

The full code for this example is in ch-15_mnist_dist_sync.py. You are encouraged to modify and explore the code with your own datasets.

For synchronous updates, the following modifications need to be made to the code:

1. The optimizer needs to be wrapped ...