How it works...
In step 2, dependencies were added for DataVec. We need to use data transformation functions in Spark just like in regular training. Transformation is a data requirement for neural networks and is not Spark-specific.
For example, we talked about LocalTransformExecutor in Chapter 2, Data Extraction, Transformation, and Loading. LocalTransformExecutor is used for DataVec transformation in non-distributed environments. SparkTransformExecutor will be used for the DataVec transformation process in Spark.
In step 4, we added dependencies for gradient sharing. Training times are faster for gradient sharing and it is designed to be scalable and fault-tolerant. Therefore, gradient sharing is preferred over parameter averaging. In ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access