O'Reilly logo
live online training icon Live Online training

Lazy Looping in Python

Making and Using Generators and Iterators

Topic: Programming
Trey Hunner

When processing large amounts of data in Python, we often reach for lists. Unfortunately, processing data using large lists results in ugly code that can be memory inefficient and slow. Python's solution to this problem is lazy looping using generators and iterators.

During this course, we'll learn a number of lazy looping techniques which will help you write more efficient and more readable Python code. We'll get practice creating generators, playing with iterators, and using generators and iterators to drastically restructure our code in a more descriptive, data-centric way. You'll get hands-on experience with Python's various lazy looping constructs and a greater appreciation for how looping works under the hood in Python.

What you'll learn-and how you can apply it

By the end of this live, hands-on, online course, you’ll understand:

  • How to break up giant for loops into smaller more readable chunks
  • When and where lazy looping techniques can be used to conserve system resources
  • How to compose generators and iterators to make readable, descriptive code

And you’ll be able to:

  • Embrace Python’s inherent laziness by working with Python’s built-in iterators directly
  • Create your own iterators for working with hundreds or millions of lines/rows/records
  • Write readable and efficient looping code that works within the limitations of Python’s iterators

This training course is for you because...

  • You’re an experienced programmer comfortable in Python
  • You work with files or database queries with thousands of lines/rows
  • You want to write looping logic that is both readable and efficient

Prerequisites

  • Comfort with Python's for loops and lists is required.
  • Comfort with list comprehensions would be helpful, but is not required.

Recommended preparation:

  • Make sure that Python 3.7+ is installed on your machine.

Recommended follow-up:

About your instructor

  • Trey Hunner helps software teams level up their Python skills through on-site Python training workshops and courses. Trey speaks at Python conferences, blogs about Python frequently, and runs Python Morsels, an online Python education service for individuals and small teams.

    Trey was formerly a director at the Python Software Foundation.

Schedule

The timeframes are only estimates and may vary according to how the class is progressing

Comprehensions (40 minutes)

  • Presentation: Comprehensions as a special purpose tool for building up new lists
  • Exercises: Refactoring for loops into list comprehensions
  • Q&A
  • Presentation: Refactoring complex loops
  • Break

Generator expressions (30 minutes)

  • Presentation: Generator expressions as a tool for saving memory/time and avoiding making new data structures
  • Exercises: Refactoring looping code to use generator expressions, rather than building up new lists just to loop over them once
  • Q&A

Generator functions (40 minutes)

  • Presentation: Demo code that loops over a file in multi-line chunks
  • Q&A
  • Presentation: using a generator function to lazily loop over a file in multi-line chunks in a memory-efficient way
  • Exercises: Creating generator functions to save time and memory
  • Q&A

The iterator protocol (40 minutes)

  • Break
  • Presentation: How generators (and Python’s for loops) work under the hood
  • Exercise: Working with the many iterator-powered looping utilities built-in to Python and creating your own iterators
  • Q&A

Lazy looping helpers (30 minutes)

  • Presentation: Python’s itertools module contains helpers for using iterators and generators to loop efficiently
  • Exercise: Using the various lazy looping helpers in the itertools module to complement our efficient generator and iterator objects
  • Q&A