Cleaning Data for Effective Data Science

Cleaning Data for Effective Data Science

by David Mertz
Released March 2021
Publisher(s): Packt Publishing
ISBN: 9781801071291

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Think about your data intelligently and ask the right questions

Key Features

  • Master data cleaning techniques necessary to perform real-world data science and machine learning tasks
  • Spot common problems with dirty data and develop flexible solutions from first principles
  • Test and refine your newly acquired skills through detailed exercises at the end of each chapter

Book Description

Data cleaning is the all-important first step to successful data science, data analysis, and machine learning. If you work with any kind of data, this book is your go-to resource, arming you with the insights and heuristics experienced data scientists had to learn the hard way.

In a light-hearted and engaging exploration of different tools, techniques, and datasets real and fictitious, Python veteran David Mertz teaches you the ins and outs of data preparation and the essential questions you should be asking of every piece of data you work with.

Using a mixture of Python, R, and common command-line tools, Cleaning Data for Effective Data Science follows the data cleaning pipeline from start to end, focusing on helping you understand the principles underlying each step of the process. You'll look at data ingestion of a vast range of tabular, hierarchical, and other data formats, impute missing values, detect unreliable data and statistical anomalies, and generate synthetic features. The long-form exercises at the end of each chapter let you get hands-on with the skills you've acquired along the way, also providing a valuable resource for academic courses.

What you will learn

  • Ingest and work with common data formats like JSON, CSV, SQL and NoSQL databases, PDF, and binary serialized data structures
  • Understand how and why we use tools such as pandas, SciPy, scikit-learn, Tidyverse, and Bash
  • Apply useful rules and heuristics for assessing data quality and detecting bias, like Benford's law and the 68-95-99.7 rule
  • Identify and handle unreliable data and outliers, examining z-score and other statistical properties
  • Impute sensible values into missing data and use sampling to fix imbalances
  • Use dimensionality reduction, quantization, one-hot encoding, and other feature engineering techniques to draw out patterns in your data
  • Work carefully with time series data, performing de-trending and interpolation

Who this book is for

This book is designed to benefit software developers, data scientists, aspiring data scientists, teachers, and students who work with data. If you want to improve your rigor in data hygiene or are looking for a refresher, this book is for you.

Basic familiarity with statistics, general concepts in machine learning, knowledge of a programming language (Python or R), and some exposure to data science are helpful.

Publisher resources

Download Example Code

Table of contents

  1. Preface
  2. PART I: Data Ingestion
    1. Tabular Formats
      1. Tidying Up
      2. CSV
        1. Sanity Checks
        2. The Good, the Bad, and the Textual Data
          1. The Bad
          2. The Good
      3. Spreadsheets Considered Harmful
      4. SQL RDBMS
        1. Massaging Data Types
        2. Repeating in R
        3. Where SQL Goes Wrong (and How to Notice It)
      5. Other Formats
        1. HDF5 and NetCDF-4
          1. Tools and Libraries
        2. SQLite
        3. Apache Parquet
      6. Data Frames
        1. Spark/Scala
        2. Pandas and Derived Wrappers
        3. Vaex
        4. Data Frames in R (Tidyverse)
        5. Data Frames in R (data.table)
        6. Bash for Fun
      7. Exercises
        1. Tidy Data from Excel
        2. Tidy Data from SQL
      8. Denouement
    2. Hierarchical Formats
      1. JSON
        1. What JSON Looks Like
        2. NaN Handling and Data Types
        3. JSON Lines
        4. GeoJSON
        5. Tidy Geography
        6. JSON Schema
      2. XML
        1. User Records
        2. Keyhole Markup Language
      3. Configuration Files
        1. INI and Flat Custom Formats
        2. TOML
        3. Yet Another Markup Language
      4. NoSQL Databases
        1. Document-Oriented Databases
          1. Missing Fields
          2. Denormalization and Its Discontents
        2. Key/Value Stores
      5. Exercises
        1. Exploring Filled Area
        2. Create a Relational Model
      6. Denouement
    3. Repurposing Data Sources
      1. Web Scraping
        1. HTML Tables
        2. Non-Tabular Data
        3. Command-Line Scraping
      2. Portable Document Format
      3. Image Formats
        1. Pixel Statistics
        2. Channel Manipulation
        3. Metadata
      4. Binary Serialized Data Structures
      5. Custom Text Formats
        1. A Structured Log
        2. Character Encodings
      6. Exercises
        1. Enhancing the NPY Parser
        2. Scraping Web Traffic
      7. Denouement
  3. PART II: The Vicissitudes of Error
    1. Anomaly Detection
      1. Missing Data
        1. SQL
        2. Hierarchical Formats
        3. Sentinels
      2. Miscoded Data
      3. Fixed Bounds
      4. Outliers
        1. Z-Score
        2. Interquartile Range
      5. Multivariate Outliers
      6. Exercises
        1. A Famous Experiment
        2. Misspelled Words
      7. Denouement
    2. Data Quality
      1. Missing Data
      2. Biasing Trends
        1. Understanding Bias
        2. Detecting Bias
        3. Comparison to Baselines
        4. Benford’s Law
      3. Class Imbalance
      4. Normalization and Scaling
        1. Applying a Machine Learning Model
        2. Scaling Techniques
        3. Factor and Sample Weighting
      5. Cyclicity and Autocorrelation
        1. Domain Knowledge Trends
        2. Discovered Cycles
      6. Bespoke Validation
        1. Collation Validation
        2. Transcription Validation
      7. Exercises
        1. Data Characterization
        2. Oversampled Polls
      8. Denouement
  4. PART III: Rectification and Creation
    1. Value Imputation
      1. Typical-Value Imputation
        1. Typical Tabular Data
        2. Locality Imputation
      2. Trend Imputation
        1. Types of Trends
        2. A Larger Coarse Time Series
          1. Understanding the Data
          2. Removing Unusable Data
          3. Imputing Consistency
          4. Interpolation
        3. Non-Temporal Trends
      3. Sampling
        1. Undersampling
        2. Oversampling
      4. Exercises
        1. Alternate Trend Imputation
        2. Balancing Multiple Features
      5. Denouement
    2. Feature Engineering
      1. Date/Time Fields
        1. Creating Datetimes
        2. Imposing Regularity
        3. Duplicated Timestamps
        4. Adding Timestamps
      2. String Fields
        1. Fuzzy Matching
        2. Explicit Categories
      3. String Vectors
        1. Decompositions
        2. Rotation and Whitening
        3. Dimensionality Reduction
        4. Visualization
      4. Quantization and Binarization
      5. One-Hot Encoding
      6. Polynomial Features
        1. Generating Synthetic Features
        2. Feature Selection
      7. Exercises
        1. Intermittent Occurrences
        2. Characterizing Levels
      8. Denouement
  5. PART IV: Ancillary Matters
    1. Closure
      1. What You Know
      2. What You Don’t Know (Yet)
    2. Glossary
  6. Why subscribe?
  7. Other Books You May Enjoy
  8. Index

Product information

  • Title: Cleaning Data for Effective Data Science
  • Author(s): David Mertz
  • Release date: March 2021
  • Publisher(s): Packt Publishing
  • ISBN: 9781801071291