Enterprise Data Workflows with Cascading

Enterprise Data Workflows with Cascading

by Paco Nathan
Released July 2013
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781449359607

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

There is an easier way to build Hadoop applications. With this hands-on book, you’ll learn how to use Cascading, the open source abstraction framework for Hadoop that lets you easily create and manage powerful enterprise-grade data processing applications—without having to learn the intricacies of MapReduce.

Working with sample apps based on Java and other JVM languages, you’ll quickly learn Cascading’s streamlined approach to data processing, data filtering, and workflow optimization. This book demonstrates how this framework can help your business extract meaningful information from large amounts of distributed data.

  • Start working on Cascading example projects right away
  • Model and analyze unstructured data in any format, from any source
  • Build and test applications with familiar constructs and reusable components
  • Work with the Scalding and Cascalog Domain-Specific Languages
  • Easily deploy applications to Hadoop, regardless of cluster location or data size
  • Build workflows that integrate several big data frameworks and processes
  • Explore common use cases for Cascading, including features and tools that support them
  • Examine a case study that uses a dataset from the Open Data Initiative

Publisher resources

View/Submit Errata

Table of contents

  1. Enterprise Data Workflows with Cascading
  2. Preface
    1. Requirements
    2. Enterprise Data Workflows
    3. Complexity, More So Than Bigness
    4. Origins of the Cascading API
    5. Using Code Examples
    6. Safari® Books Online
    7. How to Contact Us
    8. Kudos
  3. 1. Getting Started
    1. Programming Environment Setup
    2. Example 1: Simplest Possible App in Cascading
    3. Build and Run
    4. Cascading Taxonomy
    5. Example 2: The Ubiquitous Word Count
    6. Flow Diagrams
    7. Predictability at Scale
  4. 2. Extending Pipe Assemblies
    1. Example 3: Customized Operations
    2. Scrubbing Tokens
    3. Example 4: Replicated Joins
    4. Stop Words and Replicated Joins
    5. Comparing with Apache Pig
    6. Comparing with Apache Hive
  5. 3. Test-Driven Development
    1. Example 5: TF-IDF Implementation
    2. Example 6: TF-IDF with Testing
    3. A Word or Two About Testing
  6. 4. Scalding—A Scala DSL for Cascading
    1. Why Use Scalding?
    2. Getting Started with Scalding
    3. Example 3 in Scalding: Word Count with Customized Operations
    4. A Word or Two about Functional Programming
    5. Example 4 in Scalding: Replicated Joins
    6. Build Scalding Apps with Gradle
    7. Running on Amazon AWS
  7. 5. Cascalog—A Clojure DSL for Cascading
    1. Why Use Cascalog?
    2. Getting Started with Cascalog
    3. Example 1 in Cascalog: Simplest Possible App
    4. Example 4 in Cascalog: Replicated Joins
    5. Example 6 in Cascalog: TF-IDF with Testing
    6. Cascalog Technology and Uses
  8. 6. Beyond MapReduce
    1. Applications and Organizations
    2. Lingual, a DSL for ANSI SQL
      1. Using the SQL Command Shell
      2. Using the JDBC Driver
      3. Integrating with Desktop Tools
    3. Pattern, a DSL for Predictive Model Markup Language
      1. Getting Started with Pattern
      2. Predefined App for PMML
      3. Integrating Pattern into Cascading Apps
      4. Customer Experiments
      5. Technology Roadmap for Pattern
  9. 7. The Workflow Abstraction
    1. Key Insights
    2. Pattern Language
    3. Literate Programming
    4. Separation of Concerns
    5. Functional Relational Programming
    6. Enterprise vs. Start-Ups
  10. 8. Case Study: City of Palo Alto Open Data
    1. Why Open Data?
    2. City of Palo Alto
    3. Moving from Raw Sources to Data Products
    4. Calibrating Metrics for the Recommender
    5. Spatial Indexing
    6. Personalization
    7. Recommendations
    8. Build and Run
    9. Key Points of the Recommender Workflow
  11. A. Troubleshooting Workflows
    1. Build and Runtime Problems
    2. Anti-Patterns
    3. Workflow Bottlenecks
    4. Other Resources
  12. Index
  13. About the Author
  14. Colophon
  15. Copyright

Product information

  • Title: Enterprise Data Workflows with Cascading
  • Author(s): Paco Nathan
  • Release date: July 2013
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781449359607