Deep Learning at Scale

Chapter 4. Putting It All Together: Efficient Deep Learning

In Chapter 2, you read about the fundamentals and data flows of deep learning applications. In Chapter 3, you learned about the various computational units that are available today and how they enable number crunching at scale. This chapter builds on the content from the previous two chapters, demonstrating the acceleration provided by specialized computing hardware and providing some examples of how-tos. It also presents some tips and tricks for efficiently training a deep learning model on a single machine with at most one accelerated device.

There are two hands-on exercises in this chapter, one using a language model (OpenAI’s GPT-2) and the second an image classification model (EfficientNet).¹ The GPT-2 exercise allows you to explore the level of acceleration a GPU provides and dig into the details of profiling tools to understand the underlying implications. In the second hands-on example, you’ll explore building a multiclass image segmentation solution using the MIT Scene Parsing Benchmark (SceneParse150) dataset. After you go through these exercises, you’ll look at several techniques you can apply to introduce efficiency in your code. More specifically, you will learn about graph compilation, mixed-precision training, efficiencies obtained via gradient tricks, memory layout tricks, and some DataLoader tricks to manage model input pipeline overheads. This chapter culminates with a small example of a custom kernel ...