Scaling Python with Ray

Foreword

In this book, Holden Karau and Boris Lublinksy touch on the biggest trend in computing today: the growing need for scalable computing. This trend is being driven, in large part, by the proliferation of machine learning (ML) throughout many industries and the growing amount of computational resources needed to do ML in practice.

The last decade has seen significant shifts in the nature of computing. In 2012, when I first began working in ML, much of it was managed on a single laptop or server, and many practitioners were using Matlab. That year was something of an inflection point as deep learning made a splash by winning the ImageNet competition by an astounding margin. That led to a sustained trend over many years in which more and more computation on more and more data has led to better results. This trend has yet to show signs of slowing down and, if anything, has accelerated in recent years with the advent of large language models.

This shift—from small models on small data to large models on large data—has changed the practice of ML. Software engineering now plays a central role in ML, and teams and organizations that successfully leverage ML often build large in-house infrastructure teams to support the distributed systems necessary for scaling ML applications across hundreds or thousands of machines.

So at the same time that ML is growing in its capabilities and becoming more relevant for a variety of businesses, it is also becoming increasingly difficult to do ...