O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Practical Data Wrangling

Book Description

Turn your noisy data into relevant, insight-ready information by leveraging the data wrangling techniques in Python and R

About This Book

  • This easy-to-follow guide takes you through every step of the data wrangling process in the best possible way
  • Work with different types of datasets, and reshape the layout of your data to make it easier for analysis
  • Get simple examples and real-life data wrangling solutions for data pre-processing

Who This Book Is For

If you are a data scientist, data analyst, or a statistician who wants to learn how to wrangle your data for analysis in the best possible manner, this book is for you. As this book covers both R and Python, some understanding of them will be beneficial.

What You Will Learn

  • Read a csv file into python and R, and print out some statistics on the data
  • Gain knowledge of the data formats and programming structures involved in retrieving API data
  • Make effective use of regular expressions in the data wrangling process
  • Explore the tools and packages available to prepare numerical data for analysis
  • Find out how to have better control over manipulating the structure of the data
  • Create a dexterity to programmatically read, audit, correct, and shape data
  • Write and complete programs to take in, format, and output data sets

In Detail

Around 80% of time in data analysis is spent on cleaning and preparing data for analysis. This is, however, an important task, and is a prerequisite to the rest of the data analysis workflow, including visualization, analysis and reporting. Python and R are considered a popular choice of tool for data analysis, and have packages that can be best used to manipulate different kinds of data, as per your requirements. This book will show you the different data wrangling techniques, and how you can leverage the power of Python and R packages to implement them.

You’ll start by understanding the data wrangling process and get a solid foundation to work with different types of data. You’ll work with different data structures and acquire and parse data from various locations. You’ll also see how to reshape the layout of data and manipulate, summarize, and join data sets. Finally, we conclude with a quick primer on accessing and processing data from databases, conducting data exploration, and storing and retrieving data quickly using databases.

The book includes practical examples on each of these points using simple and real-world data sets to give you an easier understanding. By the end of the book, you’ll have a thorough understanding of all the data wrangling concepts and how to implement them in the best possible way.

Style and approach

This is a practical book on data wrangling designed to give you an insight into the practical application of data wrangling. It takes you through complex concepts and tasks in an accessible way, featuring information on a wide range of data wrangling techniques with Python and R

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Conventions
    5. Reader feedback
    6. Customer support
      1. Downloading the example code
      2. Downloading the color images of this book
      3. Errata
      4. Piracy
      5. Questions
  2. Programming with Data
    1. Understanding data wrangling
      1. Getting and reading data
      2. Cleaning data
      3. Shaping and structuring data
      4. Storing data
    2. The tools for data wrangling
      1. Python
      2. R
    3. Summary
  3. Introduction to Programming in Python
    1. External resources
    2. Logistical overview
      1. Installation requirements
      2. Using other learning resources
      3. Python 2 versus Python 3
    3. Running programs in python
      1. Using text editors to write and manage programs
        1. Writing the hello world program
      2. Using the terminal to run programs
        1. Running the Hello World program
      3. What if it didn't work?
    4. Data types, variables, and the Python shell
      1. Numbers - integers and floats
        1. Why integers? 
      2. Strings
      3. Booleans
      4. The print function
      5. Variables
        1. Adding to a variable
        2. Subtracting from a variable
        3. Multiplication
        4. Division
        5. Naming variables
      6. Arrays (lists, if you ask Python)
      7. Dictionaries 
    5. Compound statements
      1. Compound statement syntax and indentation level
      2. For statements and iterables
      3. If statements
        1. Else and elif clauses
      4. Functions
        1. Passing arguments to a function
        2. Returning values from a function
    6. Making annotations within programs
    7. A programmer's resources
      1. Documentation
      2. Online forums and mailing lists
    8.  Summary
  4. Reading, Exploring, and Modifying Data - Part I
    1. External resources
    2. Logistical overview
      1. Installation requirements
      2. Data
      3. File system setup
    3. Introducing a basic data wrangling work flow
    4. Introducing the JSON file format
    5. Opening and closing a file in Python using file I/O
      1. The open function and file objects
      2. File structure - best practices to store your data
      3. Opening a file
    6. Reading the contents of a file
      1. Modules in Python
      2. Parsing a JSON file using the json module
    7. Exploring the contents of a data file
      1. Extracting the core content of the data
      2. Listing out all of the variables in the data
    8. Modifying a dataset
      1. Extracting data variables from the original dataset
        1. Using a for loop to iterate over the data
        2. Using a nested for loop to iterate over the data variables
    9. Outputting the modified data to a new file
    10. Specifying input and output file names in the Terminal
      1. Specifying the filenames from the Terminal
    11. Summary
  5. Reading, Exploring, and Modifying Data - Part II
    1. Logistical overview
      1. File system setup
      2. Data
      3. Installing pandas
    2. Understanding the CSV format
    3. Introducing the CSV module
    4. Using the CSV module to read CSV data
    5. Using the CSV module to write CSV data
    6. Using the pandas module to read and process data
      1. Counting the total road length in 2011 revisited
    7. Handling non-standard CSV encoding and dialect
    8. Understanding XML
      1. XML versus JSON
    9. Using the XML module to parse XML data
      1. XPath
    10. Summary
  6. Manipulating Text Data - An Introduction to Regular Expressions
    1. Logistical overview
      1. Data
      2. File structure setup
    2. Understanding the need for pattern recognition
    3. Introducting regular expressions
      1. Writing and using a regular expression
      2. Special characters
      3. Matching whitespace
      4. Matching the start of string
      5. Matching the end of a string
      6. Matching a range of characters
      7. Matching any one of several patterns
      8. Matching a sequence instead of just one character
      9. Putting patterns together
      10. Extracting a pattern from a string
      11. The regex split() function
      12. Python regex documentation
    4. Looking for patterns
    5. Quantifying the existence of patterns
      1. Creating a regular expression to match the street address
      2. Counting the number of matches
      3. Verifying the correctness of the matches
    6. Extracting patterns
      1. Outputting the data to a new file
    7. Summary
  7. Cleaning Numerical Data - An Introduction to R and RStudio
    1. Logistical overview
      1. Data
      2. Directory structure
      3. Installing R and RStudio
    2. Introducing R and RStudio
    3. Familiarizing yourself with RStudio
      1. Running R commands
      2. Setting the working directory
      3. Reading data
      4. The R dataframe
      5. R vectors
      6. Indexing R dataframes
      7. Finding the 2011 total in R
    4. Conducting basic outlier detection and removal
    5. Handling NA values
      1. Deleting missing values
      2. Replacing missing values with a constant
      3. Imputation of missing values
    6. Variable names and contents
    7. Summary
  8. Simplifying Data Manipulation with dplyr
    1. Logistical overview
      1. Data
      2. File system setup
      3. Installing the dplyr and tibble packages
    2. Introducing dplyr
    3. Getting started with dplyr
    4. Chaining operations together
    5. Filtering the rows of a dataframe
    6. Summarizing data by category
    7. Rewriting code using dplyr
    8. Summary
  9. Getting Data from the Web
    1. Logistical overview
      1. Filesystem setup
      2. Installing the requests module
      3. Internet connection
    2. Introducing APIs
    3. Using Python to retrieve data from APIs
    4. Using URL parameters to filter the results
    5. Summary
  10. Working with Large Datasets
    1. Logistical overview 
      1. System requirements
      2. Data
      3. File system setup
      4. Installing MongoDB
      5. Planning out your time
      6. Cleaning up
    2. Understanding computer memory
    3. Understanding databases
    4. Introducing MongoDB
    5. Interfacing with MongoDB from Python
    6. Summary