Learning Ray

Chapter 5. Hyperparameter Optimization with Ray Tune

In Chapter 4 you learned how to build and run various reinforcement learning experiments. Running such experiments can be expensive, in terms of both compute resources and the time it takes to run them. This expense only gets amplified as you move on to more challenging tasks, since it is unlikely that you can just pick an algorithm out of the box and run it to get a good result. In other words, at some point you’ll need to tune the hyperparameters of your algorithms to get the best results. As we’ll see in this chapter, tuning machine learning models is hard, but Ray Tune is an excellent choice to help you tackle this task.

Ray Tune is a powerful tool for hyperparameter optimization (HPO). Not only does it work in a distributed manner by default (and works in any other Ray library discussed in this book), but it’s also one of the most feature-rich HPO libraries available. To top this off, Tune integrates with some of the most prominent HPO libraries out there, such as Hyperopt, Optuna, and many more. This makes Tune an ideal candidate for distributed HPO experiments, whether you’re coming from other libraries or starting from scratch.

In this chapter we’ll first revisit in a bit more depth why HPO is hard to do and how you could naively implement it yourself with Ray. We then teach you the core concepts of Ray Tune and how you can use it to tune the RLlib models built in the previous chapter. To wrap things up, we’ll also have ...