High Performance Python, 3rd Edition

Chapter 5. Iterators and Generators

Questions You’ll Be Able to Answer After This Chapter

• How do generators save memory?

• When is the best time to use a generator?

• How can I use itertools to create complex generator workflows?

• When is lazy evaluation beneficial, and when is it not?

When many people with experience in another language start learning Python, they are taken aback by the difference in for loop notation. That is to say, instead of writing

# Other languages
for (i=0; i<N; i++) {
    do_work(i);
}

they are introduced to a new function called range:

# Python
for i in range(N):
    do_work(i)

It seems that in the Python code sample we are calling a function, range, that creates all of the data we need for the for loop to continue. Intuitively, this can be quite a time-consuming process—if we are trying to loop over the numbers 1 through 100,000,000, then we need to spend a lot of time creating that array! However, this is where generators come into play: they essentially allow us to lazily evaluate these sorts of functions so we can have the code-readability of these special-purpose functions without the performance impacts.

Note

Technically, range is a special range type and not a generator. However, it still is useful to get at the “lazy evaluation” nature of generators, especially since most uses of this range-type parallel generators. To make them exactly the same, you can run the range through the iter function to be sure that the behavior you are seeing is generator-specific ...