Learning Ray

Chapter 6. Data Processing with Ray

Edward Oakes

In Chapter 5 you learned how to tune hyperparameters for your machine learning experiments. Of course, the key component to applying machine learning in practice is data. In this chapter we’ll explore the core set of data processing capabilities on Ray: Ray Data.

While not meant to replace more general data processing systems such as Apache Spark or Apache Hadoop, Ray Data offers basic data processing capabilities and a standard way to load, transform, and pass data to different parts of a Ray application. This enables an ecosystem of libraries on Ray to speak the same language so users can mix and match functionality in a framework-agnostic way to meet their needs.

The central component of the Ray Data ecosystem, Ray Datasets, offers the core abstractions for loading, transforming, and passing references to data in a Ray Cluster. Datasets are the “glue” that enables different libraries to interoperate on top of Ray. You’ll see this in action in “External Library Integrations”, where we show how you can do dataframe processing using the full expressiveness of the Dask API using Dask on Ray and transform the result into a dataset. The main benefits of Ray Datasets are:

Flexibility

It supports a wide range of data formats, work seamlessly with library integrations like Dask on Ray, and can be passed between Ray tasks and actors without copying data.

Performance for ML workloads

It offers important features like accelerator support, ...