O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Principles of Data Wrangling

Book Description

A key task that any aspiring data-driven organization needs to learn is data wrangling, the process of converting raw data into something truly useful. This practical guide provides business analysts with an overview of various data wrangling techniques and tools, and puts the practice of data wrangling into context by asking, "What are you trying to do and why?"

Wrangling data consumes roughly 50-80% of an analyst’s time before any kind of analysis is possible. Written by key executives at Trifacta, this book walks you through the wrangling process by exploring several factors—time, granularity, scope, and structure—that you need to consider as you begin to work with data. You’ll learn a shared language and a comprehensive understanding of data wrangling, with an emphasis on recent agile analytic processes used by many of today’s data-driven organizations.

Appreciate the importance—and the satisfaction—of wrangling data the right way.

  • Understand what kind of data is available
  • Choose which data to use and at what level of detail
  • Meaningfully combine multiple sources of data
  • Decide how to distill the results to a size and shape that can drive downstream analysis

Table of Contents

  1. Foreword
  2. 1. Introduction
    1. Magic Thresholds, PYMK, and User Growth at Facebook
  3. 2. A Data Workflow Framework
    1. How Data Flows During and Across Projects
    2. Connecting Analytic Actions to Data Movement: A Holistic Workflow Framework for Data Projects
    3. Raw Data Stage Actions: Ingest Data and Create Metadata
      1. Ingesting Known and Unknown Data
      2. Creating Metadata
    4. Refined Data Stage Actions: Create Canonical Data and Conduct Ad Hoc Analyses
      1. Designing Refined Data
      2. Refined Stage Analytical Actions
    5. Production Data Stage Actions: Create Production Data and Build Automated Systems
      1. Creating Optimized Data
      2. Designing Regular Reports and Automated Products/Services
    6. Data Wrangling within the Workflow Framework
  4. 3. The Dynamics of Data Wrangling
    1. Data Wrangling Dynamics
      1. Additional Aspects: Subsetting and Sampling
    2. Core Transformation and Profiling Actions
    3. Data Wrangling in the Workflow Framework
      1. Ingesting Data
      2. Describing Data
      3. Assessing Data Utility
      4. Designing and Building Refined Data
      5. Ad Hoc Reporting
      6. Exploratory Modeling and Forecasting
      7. Building an Optimized Dataset
      8. Regular Reporting and Building Data-Driven Products and Services
  5. 4. Profiling
    1. Overview of Profiling
    2. Individual Value Profiling: Syntactic Profiling
    3. Individual Value Profiling: Semantic Profiling
    4. Set-Based Profiling
    5. Profiling Individual Values in the Candidate Master File
      1. Syntactic Profiling in the Candidate Master File
      2. Set-Based Profiling in the Candidate Master File
  6. 5. Transformation: Structuring
    1. Overview of Structuring
    2. Intrarecord Structuring: Extracting Values
      1. Positional Extraction
      2. Pattern Extraction
      3. Complex Structure Extraction
    3. Intrarecord Structuring: Combining Multiple Record Fields
    4. Interrecord Structuring: Filtering Records and Fields
    5. Interrecord Structuring: Aggregations and Pivots
      1. Simple Aggregations
      2. Column-to-Row Pivots
      3. Row-to-Column Pivots
  7. 6. Transformation: Enriching
    1. Unions
    2. Joins
    3. Inserting Metadata
    4. Derivation of Values
      1. Generic
      2. Proprietary
  8. 7. Using Transformation to Clean Data
    1. Addressing Missing/NULL Values
    2. Addressing Invalid Values
  9. 8. Roles and Responsibilities
    1. Skills and Responsibilities
      1. Data Engineer
      2. Data Architect
      3. Data Scientist
      4. Analyst
    2. Roles Across the Data Workflow Framework
    3. Organizational Best Practices
  10. 9. Data Wrangling Tools
    1. Data Size and Infrastructure
    2. Data Structures
      1. Excel
      2. SQL
      3. Trifacta Wrangler
    3. Transformation Paradigms
      1. Excel
      2. SQL
      3. Trifacta Wrangler
    4. Choosing a Data Wrangling Tool