Scaling Python with Ray

Chapter 3. Remote Functions

You often need some form of distributed or parallel computing when building modern applications at scale. Many Python developers’ introduction to parallel computing is through the multiprocessing module. Multiprocessing is limited in its ability to handle the requirements of modern applications. These requirements include the following:

• Running the same code on multiple cores or machines

• Using tooling to handle machine and processing failures

• Efficiently handling large parameters

• Easily passing information between processes

Unlike multiprocessing, Ray’s remote functions satisfy these requirements. It’s important to note that remote doesn’t necessarily refer to a separate computer, despite its name; the function could be running on the same machine. What Ray does provide is mapping function calls to the right process on your behalf. Ray takes over distributing calls to that function instead of running in the same process. When calling remote functions, you are effectively running asynchronously on multiple cores or different machines, without having to concern yourself with how or where.

Note

Asynchronously is a fancy way of saying running multiple things at the same time without waiting on each other.

In this chapter, you will learn how to create remote functions, wait for their completion, and fetch results. Once you have the basics down, you will learn to compose remote functions together to create more complex operations. Before you go ...