book

Apache Sqoop Cookbook

by Kathleen Ting, Jarek Jarcec Cecho

July 2013

Intermediate to advanced

94 pages

1h 52m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Sqoop 2Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgmentsJarcec ThanksKathleen Thanks
Downloading and Installing SqoopProblemSolutionDiscussionInstalling JDBC DriversProblemSolutionDiscussionInstalling Specialized ConnectorsProblemSolutionDiscussionStarting SqoopProblemSolutionDiscussionGetting Help with SqoopProblemSolutionDiscussion
Transferring an Entire TableProblemSolutionDiscussionSpecifying a Target DirectoryProblemSolutionDiscussionImporting Only a Subset of DataProblemSolutionDiscussionProtecting Your PasswordProblemSolutionDiscussionUsing a File Format Other Than CSVProblemSolutionDiscussionCompressing Imported DataProblemSolutionDiscussionSpeeding Up TransfersProblemSolutionDiscussionSee AlsoOverriding Type MappingProblemSolutionDiscussionControlling ParallelismProblemSolutionDiscussionEncoding NULL ValuesProblemSolutionDiscussionSee AlsoImporting All Your TablesProblemSolutionDiscussion
Importing Only New DataProblemSolutionDiscussionIncrementally Importing Mutable DataProblemSolutionDiscussionPreserving the Last Imported ValueProblemSolutionDiscussionStoring Passwords in the MetastoreProblemSolutionDiscussionOverriding the Arguments to a Saved JobProblemSolutionDiscussionSharing the Metastore Between Sqoop ClientsProblemSolutionDiscussion
Importing Data from Two TablesProblemSolutionDiscussionUsing Custom Boundary QueriesProblemSolutionDiscussionRenaming Sqoop Job InstancesProblemSolutionDiscussionImporting Queries with Duplicated ColumnsProblemSolutionDiscussion
Transferring Data from HadoopProblemSolutionDiscussionInserting Data in BatchesProblemSolutionDiscussionExporting with All-or-Nothing SemanticsProblemSolutionDiscussionUpdating an Existing Data SetProblemSolutionDiscussionUpdating or Inserting at the Same TimeProblemSolutionDiscussionSee AlsoUsing Stored ProceduresProblemSolutionDiscussionExporting into a Subset of ColumnsProblemSolutionDiscussionEncoding the NULL Value DifferentlyProblemSolutionDiscussionSee AlsoExporting Corrupted DataProblemSolutionDiscussion
Scheduling Sqoop Jobs with OozieProblemSolutionDiscussionSpecifying Commands in OozieProblemSolutionDiscussionUsing Property Parameters in OozieProblemSolutionDiscussionInstalling JDBC Drivers in OozieProblemSolutionDiscussionSee AlsoImporting Data Directly into HiveProblemSolutionDiscussionSee AlsoUsing Partitioned Hive TablesProblemSolutionDiscussionReplacing Special Delimiters During Hive ImportProblemSolutionDiscussionUsing the Correct NULL String in HiveProblemSolutionDiscussionSee AlsoImporting Data into HBaseProblemSolutionDiscussionImporting All Rows into HBaseProblemSolutionDiscussionImproving Performance When Importing into HBaseProblemSolutionDiscussion
Overriding Imported boolean Values in PostgreSQL Direct ImportProblemSolutionDiscussionSee AlsoImporting a Table Stored in Custom Schema in PostgreSQLProblemSolutionDiscussionExporting into PostgreSQL Using pg_bulkloadProblemSolutionDiscussionSee AlsoConnecting to MySQLProblemSolutionDiscussionUsing Direct MySQL Import into HiveProblemSolutionDiscussionSee AlsoUsing the upsert Feature When Exporting into MySQLProblemSolutionDiscussionSee AlsoImporting from OracleProblemSolutionDiscussionUsing Synonyms in OracleProblemSolutionDiscussionFaster Transfers with OracleProblemSolutionDiscussionSee AlsoImporting into Avro with OraOopProblemSolutionDiscussionChoosing the Proper Connector for OracleProblemSolutionDiscussionExporting into TeradataProblemSolutionDiscussionSee AlsoUsing the Cloudera Teradata ConnectorProblemSolutionDiscussionSee AlsoUsing Long Column Names in TeradataProblemSolutionDiscussion

Content preview from Apache Sqoop Cookbook

Chapter 1. Getting Started

This chapter will guide you through the basic prerequisites of using Sqoop. You will learn how to download and install Sqoop on your computer or on any node of your Hadoop cluster. Sqoop comes with a very detailed User Guide describing all the available parameters and basic usage. Rather than repeating the guide, this book focuses on applying the parameters to real use cases and helping you to deploy and use Sqoop effectively in your environment.

Downloading and Installing Sqoop

Problem

You want to install Sqoop on your computer or on any node in your Hadoop cluster.

Solution

Sqoop supports the Linux operating system, and there are several installation options. One option is the source tarball that is provided with every release. This tarball contains only the source code of the project. You can’t use it directly and will need to first compile the sources into binary executables. For your convenience, the Sqoop community provides a binary tarball for each major supported version of Hadoop along with the source tarball.

In addition to the tarballs, there are open source projects and commercial companies that provide operating system-specific packages. One such project, Apache Bigtop, provides rpm packages for Red Hat, CentOS, SUSE, and deb packages for Ubuntu and Debian. The biggest benefit of using packages over tarballs is their seamless integration with the operating system: for example, Configuration files are stored in /etc/ and logs in /var/log.