Fluent Python, 2nd Edition

Chapter 17. Iterators, Generators, and Classic Coroutines

When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve. Any other regularity in the code is a sign, to me at least, that I’m using abstractions that aren’t powerful enough—often that I’m generating by hand the expansions of some macro that I need to write.

Paul Graham, Lisp hacker and venture capitalist¹

Iteration is fundamental to data processing: programs apply computations to data series, from pixels to nucleotides. If the data doesn’t fit in memory, we need to fetch the items lazily—one at a time and on demand. That’s what an iterator does. This chapter shows how the Iterator design pattern is built into the Python language so you never need to code it by hand.

Every standard collection in Python is iterable. An iterable is an object that provides an iterator, which Python uses to support operations like:

• for loops

• List, dict, and set comprehensions

• Unpacking assignments

• Construction of collection instances

This chapter covers the following topics:

• How Python uses the iter() built-in function to handle iterable objects

• How to implement the classic Iterator pattern in Python

• How the classic Iterator pattern can be replaced by a generator function or generator expression

• How a generator function works in detail, with line-by-line descriptions

• Leveraging the general-purpose generator functions in the standard library ...