O'Reilly logo

Data Visualization with Python and JavaScript by Kyran Dale

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 9. Cleaning Data with Pandas

The previous two chapters introduced Pandas and NumPy, the Numeric Python library it extends. Armed with basic Pandas know-how, we’re ready to start the cleaning stage of our toolchain, aiming to find and eliminate the dirty data in our scraped dataset (see Chapter 6). This chapter will also extend your Pandas knowledge, introducing new methods in a working context.

In Chapter 8, we covered the core components of Pandas: the DataFrame, a programmatic spreadsheet capable of dealing with the many different datatypes found in the real world, and its building block, the Series, a heterogeneous extension of NumPy’s homogeneous ndarray. We also covered how to read from and write to different datastores, including JSON, CSV files, MongoDB, and SQL databases. Now we’ll start to put Pandas through its paces showing how it can be used to clean dirty data. I’ll introduce the key elements of data cleaning using our dirty Nobel Prize dataset as an example.

I’ll take it slowly, introducing key Pandas concepts in a working environment. Let’s first establish why cleaning data is such an important part of a data visualizer’s work.

Coming Clean About Dirty Data

I think it’s fair to say that most people entering the field of data visualization underestimate, often by a fairly large factor, the amount of time they’re going to spend trying to make their data presentable. The fact is that getting clean datasets that are a pleasure to transform into cool visualizations ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required