book

Apache Sqoop Cookbook

by Kathleen Ting, Jarek Jarcec Cecho

July 2013

Intermediate to advanced

94 pages

1h 52m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Sqoop 2Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgmentsJarcec ThanksKathleen Thanks
Downloading and Installing SqoopProblemSolutionDiscussionInstalling JDBC DriversProblemSolutionDiscussionInstalling Specialized ConnectorsProblemSolutionDiscussionStarting SqoopProblemSolutionDiscussionGetting Help with SqoopProblemSolutionDiscussion
Transferring an Entire TableProblemSolutionDiscussionSpecifying a Target DirectoryProblemSolutionDiscussionImporting Only a Subset of DataProblemSolutionDiscussionProtecting Your PasswordProblemSolutionDiscussionUsing a File Format Other Than CSVProblemSolutionDiscussionCompressing Imported DataProblemSolutionDiscussionSpeeding Up TransfersProblemSolutionDiscussionSee AlsoOverriding Type MappingProblemSolutionDiscussionControlling ParallelismProblemSolutionDiscussionEncoding NULL ValuesProblemSolutionDiscussionSee AlsoImporting All Your TablesProblemSolutionDiscussion
Importing Only New DataProblemSolutionDiscussionIncrementally Importing Mutable DataProblemSolutionDiscussionPreserving the Last Imported ValueProblemSolutionDiscussionStoring Passwords in the MetastoreProblemSolutionDiscussionOverriding the Arguments to a Saved JobProblemSolutionDiscussionSharing the Metastore Between Sqoop ClientsProblemSolutionDiscussion
Importing Data from Two TablesProblemSolutionDiscussionUsing Custom Boundary QueriesProblemSolutionDiscussionRenaming Sqoop Job InstancesProblemSolutionDiscussionImporting Queries with Duplicated ColumnsProblemSolutionDiscussion
Transferring Data from HadoopProblemSolutionDiscussionInserting Data in BatchesProblemSolutionDiscussionExporting with All-or-Nothing SemanticsProblemSolutionDiscussionUpdating an Existing Data SetProblemSolutionDiscussionUpdating or Inserting at the Same TimeProblemSolutionDiscussionSee AlsoUsing Stored ProceduresProblemSolutionDiscussionExporting into a Subset of ColumnsProblemSolutionDiscussionEncoding the NULL Value DifferentlyProblemSolutionDiscussionSee AlsoExporting Corrupted DataProblemSolutionDiscussion
Scheduling Sqoop Jobs with OozieProblemSolutionDiscussionSpecifying Commands in OozieProblemSolutionDiscussionUsing Property Parameters in OozieProblemSolutionDiscussionInstalling JDBC Drivers in OozieProblemSolutionDiscussionSee AlsoImporting Data Directly into HiveProblemSolutionDiscussionSee AlsoUsing Partitioned Hive TablesProblemSolutionDiscussionReplacing Special Delimiters During Hive ImportProblemSolutionDiscussionUsing the Correct NULL String in HiveProblemSolutionDiscussionSee AlsoImporting Data into HBaseProblemSolutionDiscussionImporting All Rows into HBaseProblemSolutionDiscussionImproving Performance When Importing into HBaseProblemSolutionDiscussion
Overriding Imported boolean Values in PostgreSQL Direct ImportProblemSolutionDiscussionSee AlsoImporting a Table Stored in Custom Schema in PostgreSQLProblemSolutionDiscussionExporting into PostgreSQL Using pg_bulkloadProblemSolutionDiscussionSee AlsoConnecting to MySQLProblemSolutionDiscussionUsing Direct MySQL Import into HiveProblemSolutionDiscussionSee AlsoUsing the upsert Feature When Exporting into MySQLProblemSolutionDiscussionSee AlsoImporting from OracleProblemSolutionDiscussionUsing Synonyms in OracleProblemSolutionDiscussionFaster Transfers with OracleProblemSolutionDiscussionSee AlsoImporting into Avro with OraOopProblemSolutionDiscussionChoosing the Proper Connector for OracleProblemSolutionDiscussionExporting into TeradataProblemSolutionDiscussionSee AlsoUsing the Cloudera Teradata ConnectorProblemSolutionDiscussionSee AlsoUsing Long Column Names in TeradataProblemSolutionDiscussion

Content preview from Apache Sqoop Cookbook

Foreword

Aaron Kimball

San Francisco, CA

It’s been four years since, via a post to the Apache JIRA, the first version of Sqoop was released to the world as an addition to Hadoop. Since then, the project has taken several turns, most recently landing as a top-level Apache project. I’ve been amazed at how many people use this small tool for a variety of large tasks. Sqoop users have imported everything from humble test data sets to mammoth enterprise data warehouses into the Hadoop Distributed Filesystem, HDFS. Sqoop is a core member of the Hadoop ecosystem, and plug-ins are provided and supported by several major SQL and ETL vendors. And Sqoop is now part of integral ETL and processing pipelines run by some of the largest users of Hadoop.

The software industry moves in cycles. At the time of Sqoop’s origin, a major concern was in “unlocking” data stored in an organization’s RDBMS and transferring it to Hadoop. Sqoop enabled users with vast troves of information stored in existing SQL tables to use new analytic tools like MapReduce and Apache Pig. As Sqoop matures, a renewed focus on SQL-oriented analytics continues to make it relevant: systems like Cloudera Impala and Dremel-style analytic engines offer powerful distributed analytics with SQL-based languages, using the common data substrate offered by HDFS.

The variety of data sources and analytic targets presents a challenge in setting up effective data transfer pipelines. Data sources can have a variety of subtle inconsistencies: different ...