Programming Pig

Programming Pig

by Alan Gates
Released October 2011
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781449302641

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon
Start your free trial

Book description

This guide is an ideal learning tool and reference for Apache Pig, the open source engine for executing parallel data flows on Hadoop. With Pig, you can batch-process data without having to create a full-fledged application—making it easy for you to experiment with new datasets.

Programming Pig introduces new users to Pig, and provides experienced users with comprehensive coverage on key features such as the Pig Latin scripting language, the Grunt shell, and User Defined Functions (UDFs) for extending Pig. If you need to analyze terabytes of data, this book shows you how to do it efficiently with Pig.

  • Delve into Pig’s data model, including scalar and complex data types
  • Write Pig Latin scripts to sort, group, join, project, and filter your data
  • Use Grunt to work with the Hadoop Distributed File System (HDFS)
  • Build complex data processing pipelines with Pig’s macros and modularity features
  • Embed Pig Latin in Python for iterative processing and other advanced tasks
  • Create your own load and store functions to handle data formats and storage mechanisms
  • Get performance tips for running scripts on Hadoop clusters in less time

Publisher resources

View/Submit Errata

Table of contents

  1. Programming Pig
  2. Dedication
  3. Preface
    1. Data Addiction
    2. Who Should Read This Book
    3. Conventions Used in This Book
    4. Code Examples in This Book
    5. Using Code Examples
    6. Safari® Books Online
    7. How to Contact Us
    8. Acknowledgments
  4. 1. Introduction
    1. What Is Pig?
      1. Pig on Hadoop
        1. MapReduce’s hello world
      2. Pig Latin, a Parallel Dataflow Language
        1. Comparing query and dataflow languages
        2. How Pig differs from MapReduce
      3. What Is Pig Useful For?
      4. Pig Philosophy
    2. Pig’s History
  5. 2. Installing and Running Pig
    1. Downloading and Installing Pig
      1. Downloading the Pig Package from Apache
      2. Downloading Pig from Cloudera
      3. Downloading Pig Artifacts from Maven
      4. Downloading the Source
    2. Running Pig
      1. Running Pig Locally on Your Machine
      2. Running Pig on Your Hadoop Cluster
      3. Running Pig in the Cloud
      4. Command-Line and Configuration Options
      5. Return Codes
  6. 3. Grunt
    1. Entering Pig Latin Scripts in Grunt
    2. HDFS Commands in Grunt
    3. Controlling Pig from Grunt
  7. 4. Pig’s Data Model
    1. Types
      1. Scalar Types
      2. Complex Types
        1. Map
        2. Tuple
        3. Bag
      3. Nulls
    2. Schemas
      1. Casts
  8. 5. Introduction to Pig Latin
    1. Preliminary Matters
      1. Case Sensitivity
      2. Comments
    2. Input and Output
      1. Load
      2. Store
      3. Dump
    3. Relational Operations
      1. foreach
        1. Expressions in foreach
        2. UDFs in foreach
        3. Naming fields in foreach
      2. Filter
      3. Group
      4. Order by
      5. Distinct
      6. Join
      7. Limit
      8. Sample
      9. Parallel
    4. User Defined Functions
      1. Registering UDFs
        1. Registering Python UDFs
      2. define and UDFs
      3. Calling Static Java Functions
  9. 6. Advanced Pig Latin
    1. Advanced Relational Operations
      1. Advanced Features of foreach
        1. flatten
        2. Nested foreach
      2. Using Different Join Implementations
        1. Joining small to large data
        2. Joining skewed data
        3. Joining sorted data
      3. cogroup
      4. union
      5. cross
    2. Integrating Pig with Legacy Code and MapReduce
      1. stream
      2. mapreduce
    3. Nonlinear Data Flows
    4. Controlling Execution
      1. set
      2. Setting the Partitioner
    5. Pig Latin Preprocessor
      1. Parameter Substitution
      2. Macros
      3. Including Other Pig Latin Scripts
  10. 7. Developing and Testing Pig Latin Scripts
    1. Development Tools
      1. Syntax Highlighting and Checking
      2. describe
      3. explain
      4. illustrate
      5. Pig Statistics
      6. MapReduce Job Status
      7. Debugging Tips
    2. Testing Your Scripts with PigUnit
  11. 8. Making Pig Fly
    1. Writing Your Scripts to Perform Well
      1. Filter Early and Often
      2. Project Early and Often
      3. Set Up Your Joins Properly
      4. Use Multiquery When Possible
      5. Choose the Right Data Type
      6. Select the Right Level of Parallelism
    2. Writing Your UDF to Perform
    3. Tune Pig and Hadoop for Your Job
    4. Using Compression in Intermediate Results
    5. Data Layout Optimization
    6. Bad Record Handling
  12. 9. Embedding Pig Latin in Python
    1. Compile
    2. Bind
      1. Binding Multiple Sets of Variables
    3. Run
      1. Running Multiple Bindings
    4. Utility Methods
  13. 10. Writing Evaluation and Filter Functions
    1. Writing an Evaluation Function in Java
      1. Where Your UDF Will Run
      2. Evaluation Function Basics
        1. Interacting with Pig values
      3. Input and Output Schemas
      4. Error Handling and Progress Reporting
      5. Constructors and Passing Data from Frontend to Backend
        1. Loading the distributed cache
        2. UDFContext
      6. Overloading UDFs
      7. Memory Issues in Eval Funcs
    2. Algebraic Interface
    3. Accumulator Interface
    4. Python UDFs
    5. Writing Filter Functions
  14. 11. Writing Load and Store Functions
    1. Load Functions
      1. Frontend Planning Functions
        1. Determining InputFormat
        2. Determining the location
        3. Getting the casting functions
      2. Passing Information from the Frontend to the Backend
      3. Backend Data Reading
        1. Getting ready to read
        2. Reading records
      4. Additional Load Function Interfaces
        1. Loading metadata
        2. Using partitions
        3. Casting bytearrays
        4. Pushing down projections
    2. Store Functions
      1. Store Function Frontend Planning
        1. Determining OutputFormat
        2. Setting the output location
        3. Checking the schema
      2. Store Functions and UDFContext
      3. Writing Data
        1. Preparing to write
        2. Writing records
      4. Failure Cleanup
      5. Storing Metadata
  15. 12. Pig and Other Members of the Hadoop Community
    1. Pig and Hive
    2. Cascading
    3. NoSQL Databases
      1. HBase
      2. Cassandra
    4. Metadata in Hadoop
  16. A. Built-in User Defined Functions and Piggybank
    1. Built-in UDFs
      1. Built-in Load and Store Functions
      2. Built-in Evaluation and Filter Functions
        1. Built-in math UDFs
        2. Built-in aggregate UDFs
        3. Built-in chararray and bytearray UDFs
        4. Built-in complex type UDFs
        5. Built-in filter functions
        6. Miscellaneous built-in UDF
    2. Piggybank
  17. B. Overview of Hadoop
    1. MapReduce
      1. Map Phase
      2. Combiner Phase
      3. Shuffle Phase
      4. Reduce Phase
      5. Output Phase
      6. Distributed Cache
      7. Handling Failure
    2. Hadoop Distributed File System
  18. Index
  19. About the Author
  20. Colophon
  21. Copyright

Product information

  • Title: Programming Pig
  • Author(s): Alan Gates
  • Release date: October 2011
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781449302641