Principles of Data Wrangling

Chapter 4. Profiling

Overview of Profiling

We have decided to begin our discussion of data wrangling actions with profiling. This is the first action that people generally undertake when beginning each stage of a data project. Why? Because you need to understand the contents of your data before you can begin transforming or analyzing that data. Fundamentally, profiling guides data transformations.

When you’re working on a data project, you often don’t have time to look at every field of every record. Profiling is the activity that helps you know what is “in” your dataset, and allows you to validate that your transformations work as intended. Often, profiling is used to assess the quality of your data. Profiling is also a crucial aid for data transformation. You frequently need to be able to quickly determine if any records contain data that might cause problems during the transformation process. For example, if your downstream analysis expects each record in a price column to contain numbers, you don’t want to have a record that includes letters or special characters.

Profiling can encompass two slightly different views:

• Examining individual values in your dataset

• Examining a summary view across multiple values in your dataset

Each of these views can often be consumed as textual information: a list of data values, a table of summary statistics, and so on. You can also build visualizations to capture profiling information about your data.

Ultimately, individual values profiling ...