In the above code, RDD ‘wordPair’ is created from existing RDD ‘word’ using map() transforma-
tion which contains word and its starting character together.
Features of Spark RDD
•In-memory Computation: The data inside RDD are stored in memory for as long as you want
to store. Keeping the data in-memory improves the performance by an order of magnitudes.
•Lazy Evaluation: The data inside RDDs are not evaluated on the go. The changes or the
computation is performed only after an action is triggered. Thus, it limits how much work it
has to do.
•Fault Tolerance: Upon the failure of worker node, using lineage of operations, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.