Software Engineering for Data Scientists

Chapter 3. Using Data Structures Effectively

In the previous chapter, you saw how to measure the performance of your code. In this chapter, I’ll show you how your choice of data structure can affect your code’s performance, and I’ll discuss how to choose the best data structure for the problem you’re working on.

As a data scientist, when you’re writing code, you’ll need to use a variety of data structures to store your data. You will have a lot of choices for which data structure to use, and it’s likely that some are appropriate for the problem you’re working on, and others are less good choices.

...a large part of performant programming is knowing what questions you are trying to ask of your data, and picking a data structure that can answer those questions quickly.

Micha Gorelick and Ian Osvald, High Performance Python

It’s important to use the correct data structure for the problem you’re working on for two main reasons: first, the data structure is optimized for that use case, and second, useful methods are associated with it. So if you choose the correct data structure, your code performs better and is also easier to use. It also means that your code is more predictable and easier to understand.

In this chapter, I’ll go through some of the more common data structures that you’ll use when writing data science code: Python lists, tuples, dictionaries, and sets; NumPy arrays; and pandas DataFrames. I’ll describe the advantages and disadvantages of each and discuss which ones ...