book

Deep Learning with Python, Second Edition

by Francois Chollet

November 2021

Intermediate to advanced

504 pages

15h 55m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 Artificial intelligence, machine learning, and deep learning1.1.1 Artificial intelligence1.1.2 Machine learning1.1.3 Learning rules and representations from data1.1.4 The “deep” in “deep learning”1.1.5 Understanding how deep learning works, in three figures1.1.6 What deep learning has achieved so far1.1.7 Don’t believe the short-term hype1.1.8 The promise of AI1.2 Before deep learning: A brief history of machine learning1.2.1 Probabilistic modeling1.2.2 Early neural networks1.2.3 Kernel methods1.2.4 Decision trees, random forests, and gradient boosting machines1.2.5 Back to neural networks1.2.6 What makes deep learning different1.2.7 The modern machine learning landscape1.3 Why deep learning? Why now?1.3.1 Hardware1.3.2 Data1.3.3 Algorithms1.3.4 A new wave of investment1.3.5 The democratization of deep learning1.3.6 Will it last?
2.1 A first look at a neural network2.2 Data representations for neural networks2.2.1 Scalars (rank-0 tensors)2.2.2 Vectors (rank-1 tensors)2.2.3 Matrices (rank-2 tensors)2.2.4 Rank-3 and higher-rank tensors2.2.5 Key attributes2.2.6 Manipulating tensors in NumPy2.2.7 The notion of data batches2.2.8 Real-world examples of data tensors2.2.9 Vector data2.2.10 Timeseries data or sequence data2.2.11 Image data2.2.12 Video data2.3 The gears of neural networks: Tensor operations2.3.1 Element-wise operations2.3.2 Broadcasting2.3.3 Tensor product2.3.4 Tensor reshaping2.3.5 Geometric interpretation of tensor operations2.3.6 A geometric interpretation of deep learning2.4 The engine of neural networks: Gradient-based optimization2.4.1 What’s a derivative?2.4.2 Derivative of a tensor operation: The gradient2.4.3 Stochastic gradient descent2.4.4 Chaining derivatives: The Backpropagation algorithm2.5 Looking back at our first example2.5.1 Reimplementing our first example from scratch in TensorFlow2.5.2 Running one training step2.5.3 The full training loop2.5.4 Evaluating the modelSummary
3.1 What’s TensorFlow?3.2 What’s Keras?3.3 Keras and TensorFlow: A brief history3.4 Setting up a deep learning workspace3.4.1 Jupyter notebooks: The preferred way to run deep learning experiments3.4.2 Using Colaboratory3.5 First steps with TensorFlow3.5.1 Constant tensors and variables3.5.2 Tensor operations: Doing math in TensorFlow3.5.3 A second look at the GradientTape API3.5.4 An end-to-end example: A linear classifier in pure TensorFlow3.6 Anatomy of a neural network: Understanding core Keras APIs3.6.1 Layers: The building blocks of deep learning3.6.2 From layers to models3.6.3 The “compile” step: Configuring the learning process3.6.4 Picking a loss function3.6.5 Understanding the fit() method3.6.6 Monitoring loss and metrics on validation data3.6.7 Inference: Using a model after trainingSummary
4.1 Classifying movie reviews: A binary classification example4.1.1 The IMDB dataset4.1.2 Preparing the data4.1.3 Building your model4.1.4 Validating your approach4.1.5 Using a trained model to generate predictions on new data4.1.6 Further experiments4.1.7 Wrapping up4.2 Classifying newswires: A multiclass classification example4.2.1 The Reuters dataset4.2.2 Preparing the data4.2.3 Building your model4.2.4 Validating your approach4.2.5 Generating predictions on new data4.2.6 A different way to handle the labels and the loss4.2.7 The importance of having sufficiently large intermediate layers4.2.8 Further experiments4.2.9 Wrapping up4.3 Predicting house prices: A regression example4.3.1 The Boston housing price dataset4.3.2 Preparing the data4.3.3 Building your model4.3.4 Validating your approach using K-fold validation4.3.5 Generating predictions on new data4.3.6 Wrapping upSummary

5.1 Generalization: The goal of machine learning5.1.1 Underfitting and overfitting5.1.2 The nature of generalization in deep learning5.2 Evaluating machine learning models5.2.1 Training, validation, and test sets5.2.2 Beating a common-sense baseline5.2.3 Things to keep in mind about model evaluation5.3 Improving model fit5.3.1 Tuning key gradient descent parameters5.3.2 Leveraging better architecture priors5.3.3 Increasing model capacity5.4 Improving generalization5.4.1 Dataset curation5.4.2 Feature engineering5.4.3 Using early stopping5.4.4 Regularizing your modelSummary
6.1 Define the task6.1.1 Frame the problem6.1.2 Collect a dataset6.1.3 Understand your data6.1.4 Choose a measure of success6.2 Develop a model6.2.1 Prepare the data6.2.2 Choose an evaluation protocol6.2.3 Beat a baseline6.2.4 Scale up: Develop a model that overfits6.2.5 Regularize and tune your model6.3 Deploy the model6.3.1 Explain your work to stakeholders and set expectations6.3.2 Ship an inference model6.3.3 Monitor your model in the wild6.3.4 Maintain your modelSummary
7.1 A spectrum of workflows7.2 Different ways to build Keras models7.2.1 The Sequential model7.2.2 The Functional API7.2.3 Subclassing the Model class7.2.4 Mixing and matching different components7.2.5 Remember: Use the right tool for the job7.3 Using built-in training and evaluation loops7.3.1 Writing your own metrics7.3.2 Using callbacks7.3.3 Writing your own callbacks7.3.4 Monitoring and visualization with TensorBoard7.4 Writing your own training and evaluation loops7.4.1 Training versus inference7.4.2 Low-level usage of metrics7.4.3 A complete training and evaluation loop7.4.4 Make it fast with tf.function7.4.5 Leveraging fit() with a custom training loopSummary
8.1 Introduction to convnets8.1.1 The convolution operation8.1.2 The max-pooling operation8.2 Training a convnet from scratch on a small dataset8.2.1 The relevance of deep learning for small-data problems8.2.2 Downloading the data8.2.3 Building the model8.2.4 Data preprocessing8.2.5 Using data augmentation8.3 Leveraging a pretrained model8.3.1 Feature extraction with a pretrained model8.3.2 Fine-tuning a pretrained modelSummary
9.1 Three essential computer vision tasks9.2 An image segmentation example9.3 Modern convnet architecture patterns9.3.1 Modularity, hierarchy, and reuse9.3.2 Residual connections9.3.3 Batch normalization9.3.4 Depthwise separable convolutions9.3.5 Putting it together: A mini Xception-like model9.4 Interpreting what convnets learn9.4.1 Visualizing intermediate activations9.4.2 Visualizing convnet filters9.4.3 Visualizing heatmaps of class activationSummary
10.1 Different kinds of timeseries tasks10.2 A temperature-forecasting example10.2.1 Preparing the data10.2.2 A common-sense, non-machine learning baseline10.2.3 Let’s try a basic machine learning model10.2.4 Let’s try a 1D convolutional model10.2.5 A first recurrent baseline10.3 Understanding recurrent neural networks10.3.1 A recurrent layer in Keras10.4 Advanced use of recurrent neural networks10.4.1 Using recurrent dropout to fight overfitting10.4.2 Stacking recurrent layers10.4.3 Using bidirectional RNNs10.4.4 Going even furtherSummary
11.1 Natural language processing: The bird’s eye view11.2 Preparing text data11.2.1 Text standardization11.2.2 Text splitting (tokenization)11.2.3 Vocabulary indexing11.2.4 Using the TextVectorization layer11.3 Two approaches for representing groups of words: Sets and sequences11.3.1 Preparing the IMDB movie reviews data11.3.2 Processing words as a set: The bag-of-words approach11.3.3 Processing words as a sequence: The sequence model approach11.4 The Transformer architecture11.4.1 Understanding self-attention11.4.2 Multi-head attention11.4.3 The Transformer encoder11.4.4 When to use sequence models over bag-of-words models11.5 Beyond text classification: Sequence-to-sequence learning11.5.1 A machine translation example11.5.2 Sequence-to-sequence learning with RNNs11.5.3 Sequence-to-sequence learning with TransformerSummary
12.1 Text generation12.1.1 A brief history of generative deep learning for sequence generation12.1.2 How do you generate sequence data?12.1.3 The importance of the sampling strategy12.1.4 Implementing text generation with Keras12.1.5 A text-generation callback with variable-temperature sampling12.1.6 Wrapping up12.2 DeepDream12.2.1 Implementing DeepDream in Keras12.2.2 Wrapping up12.3 Neural style transfer12.3.1 The content loss12.3.2 The style loss12.3.3 Neural style transfer in Keras12.3.4 Wrapping up12.4 Generating images with variational autoencoders12.4.1 Sampling from latent spaces of images12.4.2 Concept vectors for image editing12.4.3 Variational autoencoders12.4.4 Implementing a VAE with Keras12.4.5 Wrapping up12.5 Introduction to generative adversarial networks12.5.1 A schematic GAN implementation12.5.2 A bag of tricks12.5.3 Getting our hands on the CelebA dataset12.5.4 The discriminator12.5.5 The generator12.5.6 The adversarial network12.5.7 Wrapping upSummary
13.1 Getting the most out of your models13.1.1 Hyperparameter optimization13.1.2 ensembling13.2 Scaling-up model training13.2.1 Speeding up training on GPU with mixed precision13.2.2 Multi-GPU training13.2.3 TPU trainingSummary
14.1 Key concepts in review14.1.1 Various approaches to AI14.1.2 What makes deep learning special within the field of machine learning14.1.3 How to think about deep learning14.1.4 Key enabling technologies14.1.5 The universal machine learning workflow14.1.6 Key network architectures14.1.7 The space of possibilities14.2 The limitations of deep learning14.2.1 The risk of anthropomorphizing machine learning models14.2.2 Automatons vs. intelligent agents14.2.3 Local generalization vs. extreme generalization14.2.4 The purpose of intelligence14.2.5 Climbing the spectrum of generalization14.3 Setting the course toward greater generality in AI14.3.1 On the importance of setting the right objective: The shortcut rule14.3.2 A new target14.4 Implementing intelligence: The missing ingredients14.4.1 Intelligence as sensitivity to abstract analogies14.4.2 The two poles of abstraction14.4.3 The missing half of the picture14.5 The future of deep learning14.5.1 Models as programs14.5.2 Blending together deep learning and program synthesis14.5.3 Lifelong learning and modular subroutine reuse14.5.4 The long-term vision14.6 Staying up to date in a fast-moving field14.6.1 Practice on real-world problems using Kaggle14.6.2 Read about the latest developments on arXiv14.6.3 Explore the Keras ecosystemFinal words

Content preview from Deep Learning with Python, Second Edition

11 Deep learning for text

This chapter covers

Preprocessing text data for machine learning applications
Bag-of-words approaches and sequence-modeling approaches for text processing
The Transformer architecture
Sequence-to-sequence learning

11.1 Natural language processing: The bird’s eye view

In computer science, we refer to human languages, like English or Mandarin, as “natural” languages, to distinguish them from languages that were designed for machines, like Assembly, LISP, or XML. Every machine language was designed: its starting point was a human engineer writing down a set of formal rules to describe what statements you could make in that language and what they meant. Rules came first, and people only started using the language once ...