book

Deep Learning with PyTorch

by Eli Stevens, Thomas Viehmann, Luca Pietro Giovanni Antiga

July 2020

Intermediate to advanced

520 pages

15h 29m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A roadmapAbout the codeHardware and software requirementsliveBook discussion forumOther online resourcesabout the authorsabout the cover illustration
1.1 The deep learning revolution1.2 PyTorch for deep learning1.3 Why PyTorch?1.3.1 The deep learning competitive landscape1.4 An overview of how PyTorch supports deep learning projects1.5 Hardware and software requirements1.5.1 Using Jupyter Notebooks1.6 Exercises1.7 Summary
2.1 A pretrained network that recognizes the subject of an image2.1.1 Obtaining a pretrained network for image recognition2.1.2 AlexNet2.1.3 ResNet2.1.4 Ready, set, almost run2.1.5 Run!2.2 A pretrained model that fakes it until it makes it2.2.1 The GAN game2.2.2 CycleGAN2.2.3 A network that turns horses into zebras2.3 A pretrained network that describes scenes2.3.1 NeuralTalk22.4 Torch Hub2.5 Conclusion2.6 Exercises2.7 Summary
3.1 The world as floating-point numbers3.2 Tensors: Multidimensional arrays3.2.1 From Python lists to PyTorch tensors3.2.2 Constructing our first tensors3.2.3 The essence of tensors3.3 Indexing tensors3.4 Named tensors3.5 Tensor element types3.5.1 Specifying the numeric type with dtype3.5.2 A dtype for every occasion3.5.3 Managing a tensor’s dtype attribute3.6 The tensor API3.7 Tensors: Scenic views of storage3.7.1 Indexing into storage3.7.2 Modifying stored values: In-place operations3.8 Tensor metadata: Size, offset, and stride3.8.1 Views of another tensor’s storage3.8.2 Transposing without copying3.8.3 Transposing in higher dimensions3.8.4 Contiguous tensors3.9 Moving tensors to the GPU3.9.1 Managing a tensor’s device attribute3.10 NumPy interoperability3.11 Generalized tensors are tensors, too3.12 Serializing tensors3.12.1 Serializing to HDF5 with h5py3.13 Conclusion3.14 Exercises3.15 Summary
4.1 Working with images4.1.1 Adding color channels4.1.2 Loading an image file4.1.3 Changing the layout4.1.4 Normalizing the data4.2 3D images: Volumetric data4.2.1 Loading a specialized format4.3 Representing tabular data4.3.1 Using a real-world dataset4.3.2 Loading a wine data tensor4.3.3 Representing scores4.3.4 One-hot encoding4.3.5 When to categorize4.3.6 Finding thresholds4.4 Working with time series4.4.1 Adding a time dimension4.4.2 Shaping the data by time period4.4.3 Ready for training4.5 Representing text4.5.1 Converting text to numbers4.5.2 One-hot-encoding characters4.5.3 One-hot encoding whole words4.5.4 Text embeddings4.5.5 Text embeddings as a blueprint4.6 Conclusion4.7 Exercises4.8 Summary
5.1 A timeless lesson in modeling5.2 Learning is just parameter estimation5.2.1 A hot problem5.2.2 Gathering some data5.2.3 Visualizing the data5.2.4 Choosing a linear model as a first try5.3 Less loss is what we want5.3.1 From problem back to PyTorch5.4 Down along the gradient5.4.1 Decreasing loss5.4.2 Getting analytical5.4.3 Iterating to fit the model5.4.4 Normalizing inputs5.4.5 Visualizing (again)5.5 PyTorch’s autograd: Backpropagating all things5.5.1 Computing the gradient automatically5.5.2 Optimizers a la carte5.5.3 Training, validation, and overfitting5.5.4 Autograd nits and switching it off5.6 Conclusion5.7 Exercise5.8 Summary

6.1 Artificial neurons6.1.1 Composing a multilayer network6.1.2 Understanding the error function6.1.3 All we need is activation6.1.4 More activation functions6.1.5 Choosing the best activation function6.1.6 What learning means for a neural network6.2 The PyTorch nn module6.2.1 Using __call__ rather than forward6.2.2 Returning to the linear model6.3 Finally a neural network6.3.1 Replacing the linear model6.3.2 Inspecting the parameters6.3.3 Comparing to the linear model6.4 Conclusion6.5 Exercises6.6 Summary
7.1 A dataset of tiny images7.1.1 Downloading CIFAR-107.1.2 The Dataset class7.1.3 Dataset transforms7.1.4 Normalizing data7.2 Distinguishing birds from airplanes7.2.1 Building the dataset7.2.2 A fully connected model7.2.3 Output of a classifier7.2.4 Representing the output as probabilities7.2.5 A loss for classifying7.2.6 Training the classifier7.2.7 The limits of going fully connected7.3 Conclusion7.4 Exercises7.5 Summary
8.1 The case for convolutions8.1.1 What convolutions do8.2 Convolutions in action8.2.1 Padding the boundary8.2.2 Detecting features with convolutions8.2.3 Looking further with depth and pooling8.2.4 Putting it all together for our network8.3 Subclassing nn.Module8.3.1 Our network as an nn.Module8.3.2 How PyTorch keeps track of parameters and submodules8.3.3 The functional API8.4 Training our convnet8.4.1 Measuring accuracy8.4.2 Saving and loading our model8.4.3 Training on the GPU8.5 Model design8.5.1 Adding memory capacity: Width8.5.2 Helping our model to converge and generalize: Regularization8.5.3 Going deeper to learn more complex structures: Depth8.5.4 Comparing the designs from this section8.5.5 It’s already outdated8.6 Conclusion8.7 Exercises8.8 Summary
9.1 Introduction to the use case9.2 Preparing for a large-scale project9.3 What is a CT scan, exactly?9.4 The project: An end-to-end detector for lung cancer9.4.1 Why can’t we just throw data at a neural network until it works?9.4.2 What is a nodule?9.4.3 Our data source: The LUNA Grand Challenge9.4.4 Downloading the LUNA data9.5 Conclusion9.6 Summary
10.1 Raw CT data files10.2 Parsing LUNA’s annotation data10.2.1 Training and validation sets10.2.2 Unifying our annotation and candidate data10.3 Loading individual CT scans10.3.1 Hounsfield Units10.4 Locating a nodule using the patient coordinate system10.4.1 The patient coordinate system10.4.2 CT scan shape and voxel sizes10.4.3 Converting between millimeters and voxel addresses10.4.4 Extracting a nodule from a CT scan10.5 A straightforward dataset implementation10.5.1 Caching candidate arrays with the getCtRawCandidate function10.5.2 Constructing our dataset in LunaDataset.__init__10.5.3 A training/validation split10.5.4 Rendering the data10.6 Conclusion10.7 ExercisesSummary
11.1 A foundational model and training loop11.2 The main entry point for our application11.3 Pretraining setup and initialization11.3.1 Initializing the model and optimizer11.3.2 Care and feeding of data loaders11.4 Our first-pass neural network design11.4.1 The core convolutions11.4.2 The full model11.5 Training and validating the model11.5.1 The computeBatchLoss function11.5.2 The validation loop is similar11.6 Outputting performance metrics11.6.1 The logMetrics function11.7 Running the training script11.7.1 Needed data for training11.7.2 Interlude: The enumerateWithEstimate function11.8 Evaluating the model: Getting 99.7% correct means we’re done, right?11.9 Graphing training metrics with TensorBoard11.9.1 Running TensorBoard11.9.2 Adding TensorBoard support to the metrics logging function11.10 Why isn’t the model learning to detect nodules?11.11 Conclusion11.12 Exercises11.13 Summary
12.1 High-level plan for improvement12.2 Good dogs vs. bad guys: False positives and false negatives12.3 Graphing the positives and negatives12.3.1 Recall is Roxie’s strength12.3.2 Precision is Preston’s forte12.3.3 Implementing precision and recall in logMetrics12.3.4 Our ultimate performance metric: The F1 score12.3.5 How does our model perform with our new metrics?12.4 What does an ideal dataset look like?12.4.1 Making the data look less like the actual and more like the “ideal”12.4.2 Contrasting training with a balanced LunaDataset to previous runs12.4.3 Recognizing the symptoms of overfitting12.5 Revisiting the problem of overfitting12.5.1 An overfit face-to-age prediction model12.6 Preventing overfitting with data augmentation12.6.1 Specific data augmentation techniques12.6.2 Seeing the improvement from data augmentation12.7 Conclusion12.8 Exercises12.9 Summary
13.1 Adding a second model to our project13.2 Various types of segmentation13.3 Semantic segmentation: Per-pixel classification13.3.1 The U-Net architecture13.4 Updating the model for segmentation13.4.1 Adapting an off-the-shelf model to our project13.5 Updating the dataset for segmentation13.5.1 U-Net has very specific input size requirements13.5.2 U-Net trade-offs for 3D vs. 2D data13.5.3 Building the ground truth data13.5.4 Implementing Luna2dSegmentationDataset13.5.5 Designing our training and validation data13.5.6 Implementing TrainingLuna2dSegmentationDataset13.5.7 Augmenting on the GPU13.6 Updating the training script for segmentation13.6.1 Initializing our segmentation and augmentation models13.6.2 Using the Adam optimizer13.6.3 Dice loss13.6.4 Getting images into TensorBoard13.6.5 Updating our metrics logging13.6.6 Saving our model13.7 Results13.8 Conclusion13.9 Exercises13.10 Summary
14.1 Towards the finish line14.2 Independence of the validation set14.3 Bridging CT segmentation and nodule candidate classification14.3.1 Segmentation14.3.2 Grouping voxels into nodule candidates14.3.3 Did we find a nodule? Classification to reduce false positives14.4 Quantitative validation14.5 Predicting malignancy14.5.1 Getting malignancy information14.5.2 An area under the curve baseline: Classifying by diameter14.5.3 Reusing preexisting weights: Fine-tuning14.5.4 More output in TensorBoard14.6 What we see when we diagnose14.6.1 Training, validation, and test sets14.7 What next? Additional sources of inspiration (and data)14.7.1 Preventing overfitting: Better regularization14.7.2 Refined training data14.7.3 Competition results and research papers14.8 Conclusion14.8.1 Behind the curtain14.9 Exercises14.10 Summary
15.1 Serving PyTorch models15.1.1 Our model behind a Flask server15.1.2 What we want from deployment15.1.3 Request batching15.2 Exporting models15.2.1 Interoperability beyond PyTorch with ONNX15.2.2 PyTorch’s own export: Tracing15.2.3 Our server with a traced model15.3 Interacting with the PyTorch JIT15.3.1 What to expect from moving beyond classic Python/PyTorch15.3.2 The dual nature of PyTorch as interface and backend15.3.3 TorchScript15.3.4 Scripting the gaps of traceability15.4 LibTorch: PyTorch in C++15.4.1 Running JITed models from C++15.4.2 C++ from the start: The C++ API15.5 Going mobile15.5.1 Improving efficiency: Model design and quantization15.6 Emerging technology: Enterprise serving of PyTorch models15.7 Conclusion15.8 Exercises15.9 Summary

Content preview from Deep Learning with PyTorch

front matter

foreword

When we started the PyTorch project in mid-2016, we were a band of open source hackers who met online and wanted to write better deep learning software. Two of the three authors of this book, Luca Antiga and Thomas Viehmann, were instrumental in developing PyTorch and making it the success that it is today.

Our goal with PyTorch was to build the most flexible framework possible to express deep learning algorithms. We executed with focus and had a relatively short development time to build a polished product for the developer market. This wouldn’t have been possible if we hadn’t been standing on the shoulders of giants. PyTorch derives a significant part of its codebase from the Torch7 project started in 2007 by Ronan Collobert ...