Principles of Data Wrangling

Chapter 2. A Data Workflow Framework

In this chapter, we present a framework for working with data. Our goal is to cover the most common sequences of actions that people take as they move through the process of accessing, transforming, and using their data. We’ll begin at the end of this process, and discuss the value you will get from your data.

In the introduction, we talked about near-term and long-term value. Another dimension of value to consider is how that value will be delivered into your organization. Will value be delivered directly, through systems that can take automated actions based on data as it is processed? Or will value be delivered indirectly, by empowering people in your organization to take a different course of action than they otherwise would have?

Indirect value

Data provides value to your organization by influencing people’s decisions or inspiring changes in processes. Example: risk modeling in the insurance industry.

Direct value

Data provides value to your organization by feeding automated systems. Example: Netflix’s recommendation system.

Indirect value from data has a long tradition. Entire professions are built on it: accounting, risk modeling in insurance, experimental design in medical research, and intelligence analytics. On a smaller scale, you might have used data to generate reports or interactive visualizations. These reports and visualizations both use data to deliver indirect value. How? When others view your report or visualization, they ...