Skip to Main Content
Hadoop Security
book

Hadoop Security

by Ben Spivey, Joey Echeverria
June 2015
Intermediate to advanced content levelIntermediate to advanced
340 pages
8h 43m
English
O'Reilly Media, Inc.
Content preview from Hadoop Security

Chapter 10. Securing Data Ingest

The preceding chapters have focused on securing Hadoop from a storage and data processing perspective. We’ve assumed that you have data in Hadoop and you want to secure access to it or to control how users share analytic resources, but we’ve neglected to explain how data gets into Hadoop in the first place.

There are many ways for data to be ingested into Hadoop. The simplest method is to copy files from a local filesystem (e.g., a local hard disk or an NFS mount) to HDFS using Hadoop’s put command, as shown in Example 10-1.

Example 10-1. Ingesting files from the command line
[alice@hadoop01 ~]$ hdfs dfs -put /mnt/data/sea*.json /data/raw/sea_fire_911/

While this method might work for some datasets, it’s much more common to ingest data from existing relational systems or set up flows of event- or log-oriented data. For these use cases, users use Sqoop and Flume, respectively.

Sqoop is designed to either pull data from a relational database into Hadoop or to push data from Hadoop into a remote database. In both cases, Sqoop launches a MapReduce job that does that actual data transfer. By default, Sqoop uses JDBC drivers to transport data between the map tasks and the database. This is called generic mode and it makes it easy to use Sqoop with new data stores, as the only requirement is the availability of JDBC drivers. For performance reasons, Sqoop also supports connectors that can use vendor-specific tools and interfaces to optimize the data ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Introduction to Hadoop Security

Introduction to Hadoop Security

Jeff Bean
Securing Hadoop

Securing Hadoop

Sudheesh Narayan

Publisher Resources

ISBN: 9781491900970Errata Page