Practical Real-time Data Processing and Analytics

RDD – the name says it all

Resilient—that means RDDs are fault tolerant—now the question is, if they are in-memory, how can they recover from the failures unless they are persisted, which brings back the disk latency? Well let me help you all here—the answer is RDD lineage that helps in the re-computation of missing/damaged partition data in the event of any failures.

RDD lineage is actually an RDD operator/dependency graph that captures the logical execution plan of all transformations applied to parent RDD, so that under event of failure the entire graph/subgraph could be recomputed to regenerate the corrupt RDD.

Distributed—well an RDD data resides in-memory across the different worker nodes in the cluster.

Dataset—well it's a collection ...