O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Enterprise Data Workflows with Cascading

Enterprise Data Workflows with Cascading

by Paco Nathan
Publisher: O'Reilly Media, Inc.
Release Date: July 2013
ISBN: 9781449359584
View table of contents
Start reading

Book Description

There is an easier way to build Hadoop applications. With this hands-on book, you’ll learn how to use Cascading, the open source abstraction framework for Hadoop that lets you easily create and manage powerful enterprise-grade data processing applications—without having to learn the intricacies of MapReduce.

Working with sample apps based on Java and other JVM languages, you’ll quickly learn Cascading’s streamlined approach to data processing, data filtering, and workflow optimization. This book demonstrates how this framework can help your business extract meaningful information from large amounts of distributed data.

  • Start working on Cascading example projects right away
  • Model and analyze unstructured data in any format, from any source
  • Build and test applications with familiar constructs and reusable components
  • Work with the Scalding and Cascalog Domain-Specific Languages
  • Easily deploy applications to Hadoop, regardless of cluster location or data size
  • Build workflows that integrate several big data frameworks and processes
  • Explore common use cases for Cascading, including features and tools that support them
  • Examine a case study that uses a dataset from the Open Data Initiative

Table of Contents

  1. Preface
    1. Requirements
    2. Enterprise Data Workflows
    3. Complexity, More So Than Bigness
    4. Origins of the Cascading API
    5. Using Code Examples
    6. Safari® Books Online
    7. How to Contact Us
    8. Kudos
  2. 1. Getting Started
    1. Programming Environment Setup
    2. Example 1: Simplest Possible App in Cascading
    3. Build and Run
    4. Cascading Taxonomy
    5. Example 2: The Ubiquitous Word Count
    6. Flow Diagrams
    7. Predictability at Scale
  3. 2. Extending Pipe Assemblies
    1. Example 3: Customized Operations
    2. Scrubbing Tokens
    3. Example 4: Replicated Joins
    4. Stop Words and Replicated Joins
    5. Comparing with Apache Pig
    6. Comparing with Apache Hive
  4. 3. Test-Driven Development
    1. Example 5: TF-IDF Implementation
    2. Example 6: TF-IDF with Testing
    3. A Word or Two About Testing
  5. 4. Scalding—A Scala DSL for Cascading
    1. Why Use Scalding?
    2. Getting Started with Scalding
    3. Example 3 in Scalding: Word Count with Customized Operations
    4. A Word or Two about Functional Programming
    5. Example 4 in Scalding: Replicated Joins
    6. Build Scalding Apps with Gradle
    7. Running on Amazon AWS
  6. 5. Cascalog—A Clojure DSL for Cascading
    1. Why Use Cascalog?
    2. Getting Started with Cascalog
    3. Example 1 in Cascalog: Simplest Possible App
    4. Example 4 in Cascalog: Replicated Joins
    5. Example 6 in Cascalog: TF-IDF with Testing
    6. Cascalog Technology and Uses
  7. 6. Beyond MapReduce
    1. Applications and Organizations
    2. Lingual, a DSL for ANSI SQL
      1. Using the SQL Command Shell
      2. Using the JDBC Driver
      3. Integrating with Desktop Tools
    3. Pattern, a DSL for Predictive Model Markup Language
      1. Getting Started with Pattern
      2. Predefined App for PMML
      3. Integrating Pattern into Cascading Apps
      4. Customer Experiments
      5. Technology Roadmap for Pattern
  8. 7. The Workflow Abstraction
    1. Key Insights
    2. Pattern Language
    3. Literate Programming
    4. Separation of Concerns
    5. Functional Relational Programming
    6. Enterprise vs. Start-Ups
  9. 8. Case Study: City of Palo Alto Open Data
    1. Why Open Data?
    2. City of Palo Alto
    3. Moving from Raw Sources to Data Products
    4. Calibrating Metrics for the Recommender
    5. Spatial Indexing
    6. Personalization
    7. Recommendations
    8. Build and Run
    9. Key Points of the Recommender Workflow
  10. A. Troubleshooting Workflows
    1. Build and Runtime Problems
    2. Anti-Patterns
    3. Workflow Bottlenecks
    4. Other Resources
  11. Index
  12. About the Author
  13. Colophon
  14. Copyright