High Performance Python, 2nd Edition

Chapter 5. Iterators and Generators

Questions You’ll Be Able to Answer After This Chapter

• How do generators save memory?

• When is the best time to use a generator?

• How can I use itertools to create complex generator workflows?

• When is lazy evaluation beneficial, and when is it not?

When many people with experience in another language start learning Python, they are taken aback by the difference in for loop notation. That is to say, instead of writing

# Other languages
for (i=0; i<N; i++) {
    do_work(i);
}

they are introduced to a new function called range:

# Python
for i in range(N):
    do_work(i)

It seems that in the Python code sample we are calling a function, range, which creates all of the data we need for the for loop to continue. Intuitively, this can be quite a time-consuming process—if we are trying to loop over the numbers 1 through 100,000,000, then we need to spend a lot of time creating that array! However, this is where generators come into play: they essentially allow us to lazily evaluate these sorts of functions so we can have the code-readability of these special-purpose functions without the performance impacts.

To understand this concept, let’s implement a function that calculates several Fibonacci numbers both by filling a list and by using a generator:

def fibonacci_list(num_items):
    numbers = []
    a, b = 0, 1
    while len(numbers) < num_items:
        numbers.append(a)
        a, b = b, a+b
    return numbers

def fibonacci_gen(num_items):
    a, b = 0, 1
    while num_items ...