March 2017
Beginner to intermediate
356 pages
7h 11m
English
Spark being an MPP environment generally does not provide a shared state as the code is executed in parallel on a remote cluster node. Separate copies of data and variables are generally used during the map() or reduce() phases, and providing an ability to have a read-write shared variable across multiple executing tasks would be grossly inefficient. Spark, however, provides two types of shared variables:
Broadcast variables - Read-only variables cached on each machineAccumulators - Variables that can be added through associative and commutative propertyLargescale data movement is often a major factor in negatively affecting performance in MPP environments and hence every care is taken to reduce data movement ...
Read now
Unlock full access