Scala and Spark for Big Data Analytics

NewHadoopRDD

NewHadoopRDD provides core functionality for reading data stored in HDFS, HBase tables, Amazon S3 using the new MapReduce API from Hadoop 2.x libraries.NewHadoopRDD can read from many different formats thus is used to interact with several external systems.

Prior to NewHadoopRDD, HadoopRDD was the only available option which used the old MapReduce API from Hadoop 1.x

class NewHadoopRDD[K, V]( sc : SparkContext, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], @transient private val _conf: Configuration)extends RDD[(K, V)]

As seen in the preceding code snippet, NewHadoopRDD takes an input format class, a key class, and a value class. Let's look at examples of NewHadoopRDD.

The simplest ...