Fluent Python, 2nd Edition

Chapter 20. Concurrent Executors

The people bashing threads are typically system programmers which have in mind use cases that the typical application programmer will never encounter in her life. [...] In 99% of the use cases an application programmer is likely to run into, the simple pattern of spawning a bunch of independent threads and collecting the results in a queue is everything one needs to know.

Michele Simionato, Python deep thinker¹

This chapter focuses on the concurrent.futures.Executor classes that encapsulate the pattern of “spawning a bunch of independent threads and collecting the results in a queue,” described by Michele Simionato. The concurrent executors make this pattern almost trivial to use, not only with threads but also with processes—useful for compute-intensive tasks.

Here I also introduce the concept of futures—objects representing the asynchronous execution of an operation, similar to JavaScript promises. This primitive idea is the foundation not only of concurrent.futures but also of the asyncio package, the subject of Chapter 21.

What’s New in This Chapter

I renamed the chapter from “Concurrency with Futures” to “Concurrent Executors” because the executors are the most important high-level feature covered here. Futures are low-level objects, focused on in “Where Are the Futures?”, but mostly invisible in the rest of the chapter.

All the HTTP client examples now use the new HTTPX library, which provides synchronous and asynchronous APIs.

The setup ...