PyTorch Pocket Reference

Chapter 5. Customizing PyTorch

Up until now, you have been using built-in PyTorch classes, functions, and libraries to design and train various predefined models, model layers, and activation functions. But what if you have a novel idea or you’re conducting cutting-edge deep learning research? Perhaps you’ve invented a totally new layer architecture or activation function. Maybe you’ve developed a new optimization algorithm or a special loss function that no one’s ever seen before.

In this chapter, I’ll show you how to create your own custom deep learning components and algorithms in PyTorch. We’ll begin by exploring how to create custom layers and activation functions, and then we’ll see how to combine these components into custom model architectures. Next, I’ll show you how to create your own loss functions and optimizer algorithms. Finally, we’ll look at how to create custom loops for training, validation, and testing.

PyTorch offers flexibility: you can extend an existing library or you can combine your customizations into your own library or package. By creating custom components you can solve new deep learning problems, speed up training, and discover innovative ways to perform deep learning.

Let’s get started by creating some custom deep learning layers and activation functions.

Custom Layers and Activations

PyTorch offers an extensive set of built-in layers and activation functions. However, what makes PyTorch so popular, especially in the research community, is how easy ...