High Performance Python

Chapter 5. Iterators and Generators

Questions You’ll Be Able to Answer After This Chapter

• How do generators save memory?
• When is the best time to use a generator?
• How can I use itertools to create complex generator workflows?
• When is lazy evaluation beneficial, and when is it not?

When many people with experience in another language start learning Python, they are taken aback by the difference in for loop notation. That is to say, instead of writing:

# Other languages
for (i=0; i<N; i++) {
    do_work(i);
}

they are instead introduced to a new function called range or xrange:

# Python
for i in range(N):
    do_work(i)

These two functions provide insight into the paradigm of programming using generators. In order to fully understand generators, let us first make simple implementations of the range and xrange functions:

def range(start, stop, step=1):
    numbers = []
    while start < stop:
        numbers.append(start)
        start += step
    return numbers

def xrange(start, stop, step=1):
    while start < stop:
        yield start # 
        start += step

for i in range(1,10000):
    pass

for i in xrange(1,10000):
    pass

This function will yield many values instead of returning one value. This turns this regular-looking function into a generator that can be repeatedly polled for the next available value.

The first thing to note is that the ...