Getting Started with Impala

Introduction

Cloudera Impala is an open source project that opens up the Apache Hadoop software stack to a wide audience of database analysts, users, and developers. The Impala massively parallel processing (MPP) engine makes SQL queries of Hadoop data simple enough to be accessible to analysts familiar with SQL and to users of business intelligence tools, and it’s fast enough to be used for interactive exploration and experimentation.

From the ground up, the Impala software is written for high performance of SQL queries distributed across clusters of connected machines.

Who Is This Book For?

This book is intended for a broad audience of users from a variety of database, data warehousing, or Big Data backgrounds. It assumes that you’re experienced enough with SQL not to need explanations for familiar statements such as CREATE TABLE, SELECT, INSERT, and their major clauses. Linux experience is a plus. Experience with the Apache Hadoop software stack is useful but not required.

This book points out instances where some aspect of Impala architecture or usage might be new to people who are experienced with databases but not the Apache Hadoop software stack.

The SQL examples in this book start from a simple base for easy comprehension, then build toward best practices that demonstrate high performance and scalability.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user. This style is also used to emphasize the names of SQL statements within paragraphs.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/oreillymedia/get-started-impala.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Getting Started with Impala by John Russell (O’Reilly). Copyright 2016 Cloudera, Inc., 978-1-491-90577-7.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals.

Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/get-started-impala.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Content Updates

March 30, 2016

This update to the first edition includes details and examples for the following new features, added during the Impala 2.0 through 2.4 releases:

• Subqueries in the WHERE clause. See “Tutorial: Subqueries”.

• Analytic functions. See “Tutorial: Analytic Functions”.

• Incremental statistics. See “Keeping Statistics Up to Date for Partitioned Tables”.

• Complex types. See “Tutorial: Complex Types”.

• Submission of Impala to the Apache Incubator. The official name is no longer Cloudera Impala; now it is Apache Impala (incubating). Although Impala was always Apache-licensed, now it has been submitted as an Apache project and is beginning to use Apache Software Foundation (ASF) infrastructure and processes. See the incubator website, where you can find the wiki, mailing lists, JIRA issue tracker, and git repository.

• Some notes about Impala integration with Kudu (currently, still in a pre-release phase). The Kudu integration will add UPDATE and DELETE statements to the Impala repertoire, for Kudu tables only. At the moment, you run Kudu alongside a fork of Impala that has the extra statements and other Kudu support.

Time Marches On, and So Do Release Numbers

Traditionally, this book has referred to Impala releases by their original numbering scheme in the 1.x and 2.x series. Since Impala 2.0, tight Impala integration with the Cloudera CDH 5 distribution has made the Impala release number synonymous with specific CDH release numbers. Under the governance of the Apache Software Foundation, the Impala release numbers will likely become prominent again, as the reference point for contributors and packagers. For your convenience, here is a quick reference for the corresponding levels:

• Impala 2.0 = CDH 5.2 (subquery enhancements, analytic functions, VARCHAR and CHAR types, smaller Parquet block size, GRANT and REVOKE)

• Impala 2.1 = CDH 5.3 (COMPUTE INCREMENTAL STATS for partitioned tables, small-query optimization, stream decompression for gzipped text)

• Impala 2.2 = CDH 5.4 (automatic log rotation, beta support for Amazon S3, lots of TIMESTAMP improvements)

• Impala 2.3 = CDH 5.5 (complex data types STRUCT, ARRAY, and MAP, TRUNCATE TABLE, optimizations for non-Impala Parquet files, lots of new built-in functions)

• Impala 2.4 = CDH 5.6 (support for the EMC DSSD storage appliance; otherwise, the same as Impala 2.3 / CDH 5.5)

Acknowledgments

I have to start by acknowledging the vision and execution of the Impala development team, led by Marcel Kornacker. I have learned a lot from them—especially Justin Erickson, Alex Behm, Lenni Kuff, Alan Choi, and Nong Li—that has made it into this book. Thanks to all the Impala team members and to Gwen Shapira, Mark Grover, Kate Ting, and Uri Laserson for their feedback and insights on my drafts.

Going a little further back, I’ve been lucky to be able to consult and collaborate with really good individuals and teams at each stage and transition in my career. Thanks to James Hamilton who convinced me to switch from programming languages to the database track all those years ago at IBM. Thanks to the late Mark Townsend at Oracle for many insights about the database industry. Thanks to Ken Jacobs who helped me switch into the open source group at Oracle, and the InnoDB team under Calvin Sun and later Sunny Bains for being great to work with and teaching me database internals. Thanks to Mike Olson and Justin Kestelyn at Cloudera for showing me the right way for a small company to tackle the enterprise software market, and to do developer and community outreach. Thanks to Paul Battaglia, Jolly Chen, and Frank Liva for building and supporting the Cloudera technical publications department.

Last but not least, this book would not be possible if not for my wonderful and supportive wife, Lotus Goldstein.