Hadoop: Data Processing and Modelling

Data confidentiality in Hadoop

Hadoop is a distributed system. All distributed systems are interconnected via a network. Networks are vulnerable to malicious sniffing of data. Data at rest can also be read if they are not protected via encryption.

Data confidentiality for data at rest is delegated to the OS that hosts the DataNode. Most modern OSes provide encryption schemes to protect data on disks under their purview. In this section, we will look at confidentiality over the wire and how to enable encryption when data is in transit.

HTTPS and encrypted shuffle

Encrypted shuffle is a feature that facilitates data confidentiality in the shuffle process. To recap, the shuffle step is where data moves from the Map to the Reduce tasks in a MapReduce ...