Elegant SciPy

Chapter 8. Big Data in Little Laptop with Toolz

GRACIE: A knife? The guy’s twelve feet tall! JACK: Seven. Hey, don’t worry, I think I can handle him.

Jack Burton, Big Trouble in Little China

Streaming is not a SciPy feature per se, but rather an approach that allows us to efficiently process large datasets, like those often seen in science. The Python language contains some useful primitives for streaming data processing, and these can be combined with Matt Rocklin’s Toolz library to generate elegant, concise code that is extremely memory-efficient. In this chapter, we will show you how to apply these streaming concepts to enable you to handle much larger datasets than can fit in your computer’s RAM.

You have probably already done some streaming, perhaps without thinking about it in these terms. The simplest form is probably iterating through lines in a files, processing each line without ever reading the entire file into memory. For example, a loop like this to calculate the mean of each row and sum them:

import numpy as np
with open('data/expr.tsv') as f:
    sum_of_means = 0
    for line in f:
        sum_of_means += np.mean(np.fromstring(line, dtype=int, sep='\t'))
print(sum_of_means)

1463.0

This strategy works really well for cases where your problem can be neatly solved with by-row processing. But things can quickly get out of hand when your code becomes more sophisticated.

In streaming programs, a function processes some of the input data, returns the processed chunk, then, while ...