Programming Elastic MapReduce

Programming Elastic MapReduce

by Kevin Schmidt, Christopher Phillips
Released December 2013
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781449363628

Explore a preview version of Programming Elastic MapReduce right now.

O’Reilly members get unlimited access to live online training experiences, plus books, videos, and digital content from 200+ publishers.

Buy on Amazon Buy from O’Reilly
Start your free trial

Book description

Although you don’t need a large computing infrastructure to process massive amounts of data with Apache Hadoop, it can still be difficult to get started. This practical guide shows you how to quickly launch data analysis projects in the cloud by using Amazon Elastic MapReduce (EMR), the hosted Hadoop framework in Amazon Web Services (AWS).

Authors Kevin Schmidt and Christopher Phillips demonstrate best practices for using EMR and various AWS and Apache technologies by walking you through the construction of a sample MapReduce log analysis application. Using code samples and example configurations, you’ll learn how to assemble the building blocks necessary to solve your biggest data analysis problems.

  • Get an overview of the AWS and Apache software tools used in large-scale data analysis
  • Go through the process of executing a Job Flow with a simple log analyzer
  • Discover useful MapReduce patterns for filtering and analyzing data sets
  • Use Apache Hive and Pig instead of Java to build a MapReduce Job Flow
  • Learn the basics for using Amazon EMR to run machine learning algorithms
  • Develop a project cost model for using Amazon EMR and other AWS tools

Table of contents

  1. Preface
    1. What Is AWS?
    2. What’s in This Book?
    3. Sign Up for AWS
    4. Code Samples in This Book
    5. Conventions Used in This Book
    6. Using Code Examples
    7. Safari® Books Online
    8. How to Contact Us
    9. Acknowledgments
  2. 1. Introduction to Amazon Elastic MapReduce
    1. Amazon Web Services Used in This Book
    2. Amazon Elastic MapReduce
    3. Amazon EMR and the Hadoop Ecosystem
    4. Amazon Elastic MapReduce Versus Traditional Hadoop Installs
      1. Data Locality
      2. Hardware
      3. Complexity
    5. Application Building Blocks
  3. 2. Data Collection and Data Analysis with AWS
    1. Log Analysis Application
    2. Log Messages as a Data Set for Analytics
    3. Understanding MapReduce
    4. Collection Stage
    5. Simulating Syslog Data
      1. Generating Logs with Bash
      2. Moving Data to S3 Storage
      3. All Roads Lead to S3
    6. Developing a MapReduce Application
    7. Custom JAR MapReduce Job
    8. Running an Amazon EMR Cluster
    9. Viewing Our Results
    10. Debugging a Job Flow
      1. Running Our Job Flow with Debugging
      2. Reviewing Job Flow Log Structure
      3. Debug Through the Amazon EMR Console
    11. Our Application and Real-World Uses
  4. 3. Data Filtering Design Patterns and Scheduling Work
    1. Extending the Application Example
    2. Understanding Web Server Logs
    3. Finding Errors in the Web Logs Using Data Filtering
      1. Mapper Code
      2. Reducer Code
      3. Driver Code
      4. Running the MapReduce Filter Job
      5. Analyzing the Results
    4. Building Summary Counts in Data Sets
      1. Mapper Code
      2. Reducer Code
      3. Analyzing the Filtered Counts Job
    5. Job Flow Scheduling
      1. Scheduling with the CLI
    6. Scheduling with AWS Data Pipeline
      1. Creating a Pipeline
      2. Adding Data Nodes
      3. Adding Activities
      4. Scheduling Pipelines
      5. Reviewing Pipeline Status
      6. AWS Pipeline Costs
    7. Real-World Uses
  5. 4. Data Analysis with Hive and Pig in Amazon EMR
    1. Amazon Job Flow Technologies
    2. What Is Pig?
    3. Utilizing Pig in Amazon EMR
      1. Connecting to the Master Node
      2. Pig Latin Primer
        1. LOAD
        2. STORE
        3. DUMP
        4. ILLUSTRATE
        5. FOREACH
        6. FILTER
        7. GROUP
      3. Exploring Data with Pig Latin
      4. Running Pig Scripts in Amazon EMR
    4. What Is Hive?
    5. Utilizing Hive in Amazon EMR
      1. Hive Primer
        1. SerDe
        2. CREATE TABLE
        3. INSERT
      2. Exploring Data with Hive
      3. Running Hive Scripts in Amazon EMR
      4. Finding the Top 10 with Hive
    6. Our Application with Hive and Pig
  6. 5. Machine Learning Using EMR
    1. A Quick Tour of Machine Learning
    2. Python and EMR
      1. Why Python?
      2. The Input Data
      3. The Mapper
      4. The Reducer
      5. Putting It All Together
      6. What About Java?
    3. What’s Next?
  7. 6. Planning AWS Projects and Managing Costs
    1. Developing a Project Cost Model
      1. Software Licensing
      2. AWS and Cloud Licensing
      3. Private Data Center and AWS Cost Comparisons
      4. Cost Calculations on an Example Application
    2. Optimizing AWS Resources to Reduce Project Costs
      1. Amazon Regions
      2. Amazon Availability Zones
      3. EC2 and EMR Costs with On Demand, Reserve, and Spot Instances
      4. Reserve Instances
      5. Spot Instances
      6. Reducing AWS Project Costs
        1. EMR and EC2 usage billed by the hour
        2. Cost efficiencies with reserved and spot instances
        3. Project storage costs
        4. Data life cycles
    3. Amazon Tools for Estimating Your Project Costs
  8. A. Amazon Web Services Resources and Tools
    1. Amazon AWS Online Resources
    2. Amazon AWS Cost Estimation Tools
    3. AWS Best Practices and Architecture
    4. Amazon EMR Distributions
  9. B. Cloud Computing, Amazon Web Services, and Their Impacts
    1. AWS Service Delivery Models
      1. Platform as a Service
      2. Infrastructure as a Service
      3. Storage as a Service
    2. Performance
    3. Elasticity and Growth
      1. Fixed Capacity
      2. Variable Capacity
    4. Security
      1. Security Is a Shared Responsibility
      2. Data Security in Elastic MapReduce
    5. Uptime and Availability
  10. C. Installation and Setup
    1. Prerequisites
    2. Installing Hadoop
    3. Building MapReduce Applications
    4. Running MapReduce Applications Locally
    5. Installing Pig
    6. Installing Hive
  11. Index
  12. Colophon
  13. Copyright

Product information

  • Title: Programming Elastic MapReduce
  • Author(s): Kevin Schmidt, Christopher Phillips
  • Release date: December 2013
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781449363628