book

Deep Learning with Python

by Francois Chollet

December 2017

Intermediate to advanced

384 pages

11h 7m

English

Manning Publications

Read now

Unlock full access

François Chollet
Who should read this bookRoadmapSoftware/hardware requirementsSource code

1.1. Artificial intelligence, machine learning, and deep learning1.1.1. Artificial intelligence1.1.2. Machine learning1.1.3. Learning representations from data1.1.4. The “deep” in deep learning1.1.5. Understanding how deep learning works, in three figures1.1.6. What deep learning has achieved so far1.1.7. Don’t believe the short-term hype1.1.8. The promise of AI1.2. Before deep learning: a brief history of machine learning1.2.1. Probabilistic modeling1.2.2. Early neural networks1.2.3. Kernel methods1.2.4. Decision trees, random forests, and gradient boosting machines1.2.5. Back to neural networks1.2.6. What makes deep learning different1.2.7. The modern machine-learning landscape1.3. Why deep learning? Why now?1.3.1. Hardware1.3.2. Data1.3.3. Algorithms1.3.4. A new wave of investment1.3.5. The democratization of deep learning1.3.6. Will it last?
2.1. A first look at a neural network2.2. Data representations for neural networks2.2.1. Scalars (0D tensors)2.2.2. Vectors (1D tensors)2.2.3. Matrices (2D tensors)2.2.4. 3D tensors and higher-dimensional tensors2.2.5. Key attributes2.2.6. Manipulating tensors in Numpy2.2.7. The notion of data batches2.2.8. Real-world examples of data tensors2.2.9. Vector data2.2.10. Timeseries data or sequence data2.2.11. Image data2.2.12. Video data2.3. The gears of neural networks: tensor operations2.3.1. Element-wise operations2.3.2. Broadcasting2.3.3. Tensor dot2.3.4. Tensor reshaping2.3.5. Geometric interpretation of tensor operations2.3.6. A geometric interpretation of deep learning2.4. The engine of neural networks: gradient-based optimization2.4.1. What’s a derivative?2.4.2. Derivative of a tensor operation: the gradient2.4.3. Stochastic gradient descent2.4.4. Chaining derivatives: the Backpropagation algorithm2.5. Looking back at our first example
3.1. Anatomy of a neural network3.1.1. Layers: the building blocks of deep learning3.1.2. Models: networks of layers3.1.3. Loss functions and optimizers: keys to configuring the learning process3.2. Introduction to Keras3.2.1. Keras, TensorFlow, Theano, and CNTK3.2.2. Developing with Keras: a quick overview3.3. Setting up a deep-learning workstation3.3.1. Jupyter notebooks: the preferred way to run deep-learning experiments3.3.2. Getting Keras running: two options3.3.3. Running deep-learning jobs in the cloud: pros and cons3.3.4. What is the best GPU for deep learning?3.4. Classifying movie reviews: a binary classification example3.4.1. The IMDB dataset3.4.2. Preparing the data3.4.3. Building your network3.4.4. Validating your approach3.4.5. Using a trained network to generate predictions on new data3.4.6. Further experiments3.4.7. Wrapping up3.5. Classifying newswires: a multiclass classification example3.5.1. The Reuters dataset3.5.2. Preparing the data3.5.3. Building your network3.5.4. Validating your approach3.5.5. Generating predictions on new data3.5.6. A different way to handle the labels and the loss3.5.7. The importance of having sufficiently large intermediate layers3.5.8. Further experiments3.5.9. Wrapping up3.6. Predicting house prices: a regression example3.6.1. The Boston Housing Price dataset3.6.2. Preparing the data3.6.3. Building your network3.6.4. Validating your approach using K-fold validation3.6.5. Wrapping up
4.1. Four branches of machine learning4.1.1. Supervised learning4.1.2. Unsupervised learning4.1.3. Self-supervised learning4.1.4. Reinforcement learning4.2. Evaluating machine-learning models4.2.1. Training, validation, and test sets4.2.2. Things to keep in mind4.3. Data preprocessing, feature engineering, and feature learning4.3.1. Data preprocessing for neural networks4.3.2. Feature engineering4.4. Overfitting and underfitting4.4.1. Reducing the network’s size4.4.2. Adding weight regularization4.4.3. Adding dropout4.5. The universal workflow of machine learning4.5.1. Defining the problem and assembling a dataset4.5.2. Choosing a measure of success4.5.3. Deciding on an evaluation protocol4.5.4. Preparing your data4.5.5. Developing a model that does better than a baseline4.5.6. Scaling up: developing a model that overfits4.5.7. Regularizing your model and tuning your hyperparameters
5.1. Introduction to convnets5.1.1. The convolution operation5.1.2. The max-pooling operation5.2. Training a convnet from scratch on a small dataset5.2.1. The relevance of deep learning for small-data problems5.2.2. Downloading the data5.2.3. Building your network5.2.4. Data preprocessing5.2.5. Using data augmentation5.3. Using a pretrained convnet5.3.1. Feature extraction5.3.2. Fine-tuning5.3.3. Wrapping up5.4. Visualizing what convnets learn5.4.1. Visualizing intermediate activations5.4.2. Visualizing convnet filters5.4.3. Visualizing heatmaps of class activation
6.1. Working with text data6.1.1. One-hot encoding of words and characters6.1.2. Using word embeddings6.1.3. Putting it all together: from raw text to word embeddings6.1.4. Wrapping up6.2. Understanding recurrent neural networks6.2.1. A recurrent layer in Keras6.2.2. Understanding the LSTM and GRU layers6.2.3. A concrete LSTM example in Keras6.2.4. Wrapping up6.3. Advanced use of recurrent neural networks6.3.1. A temperature-forecasting problem6.3.2. Preparing the data6.3.3. A common-sense, non-machine-learning baseline6.3.4. A basic machine-learning approach6.3.5. A first recurrent baseline6.3.6. Using recurrent dropout to fight overfitting6.3.7. Stacking recurrent layers6.3.8. Using bidirectional RNNs6.3.9. Going even further6.3.10. Wrapping up6.4. Sequence processing with convnets6.4.1. Understanding 1D convolution for sequence data6.4.2. 1D pooling for sequence data6.4.3. Implementing a 1D convnet6.4.4. Combining CNNs and RNNs to process long sequences6.4.5. Wrapping up
7.1. Going beyond the Sequential model: the Keras functional API7.1.1. Introduction to the functional API7.1.2. Multi-input models7.1.3. Multi-output models7.1.4. Directed acyclic graphs of layers7.1.5. Layer weight sharing7.1.6. Models as layers7.1.7. Wrapping up7.2. Inspecting and monitoring deep-learning models using Keras callba- acks and TensorBoard7.2.1. Using callbacks to act on a model during training7.2.2. Introduction to TensorBoard: the TensorFlow visualization framework7.2.3. Wrapping up7.3. Getting the most out of your models7.3.1. Advanced architecture patterns7.3.2. Hyperparameter optimization7.3.3. Model ensembling7.3.4. Wrapping up
8.1. Text generation with LSTM8.1.1. A brief history of generative recurrent networks8.1.2. How do you generate sequence data?8.1.3. The importance of the sampling strategy8.1.4. Implementing character-level LSTM text generation8.1.5. Wrapping up8.2. DeepDream8.2.1. Implementing DeepDream in Keras8.2.2. Wrapping up8.3. Neural style transfer8.3.1. The content loss8.3.2. The style loss8.3.3. Neural style transfer in Keras8.3.4. Wrapping up8.4. Generating images with variational autoencoders8.4.1. Sampling from latent spaces of images8.4.2. Concept vectors for image editing8.4.3. Variational autoencoders8.4.4. Wrapping up8.5. Introduction to generative adversarial networks8.5.1. A schematic GAN implementation8.5.2. A bag of tricks8.5.3. The generator8.5.4. The discriminator8.5.5. The adversarial network8.5.6. How to train your DCGAN8.5.7. Wrapping up
9.1. Key concepts in review9.1.1. Various approaches to AI9.1.2. What makes deep learning special within the field of machine learning9.1.3. How to think about deep learning9.1.4. Key enabling technologies9.1.5. The universal machine-learning workflow9.1.6. Key network architectures9.1.7. The space of possibilities9.2. The limitations of deep learning9.2.1. The risk of anthropomorphizing machine-learning models9.2.2. Local generalization vs. extreme generalization9.2.3. Wrapping up9.3. The future of deep learning9.3.1. Models as programs9.3.2. Beyond backpropagation and differentiable layers9.3.3. Automated machine learning9.3.4. Lifelong learning and modular subroutine reuse9.3.5. The long-term vision9.4. Staying up to date in a fast-moving field9.4.1. Practice on real-world problems using Kaggle9.4.2. Read about the latest developments on arXiv9.4.3. Explore the Keras ecosystem9.5. Final words
A.1. Installing the Python scientific suiteA.2. Setting up GPU supportA.3. Installing Theano (optional)A.4. Installing Keras
B.1. What are Jupyter notebooks? Why run Jupyter notebooks on AWS GPUs?B.2. Why would you not want to use Jupyter on AWS for deep learning?B.3. Setting up an AWS GPU instanceB.3.1. Configuring JupyterB.4. Installing KerasB.5. Setting up local port forwardingB.6. Using Jupyter from your local browser

Content preview from Deep Learning with Python

Chapter 6. Deep learning for text and sequences

This chapter covers

Preprocessing text data into useful representations
Working with recurrent neural networks
Using 1D convnets for sequence processing

This chapter explores deep-learning models that can process text (understood as sequences of words or sequences of characters), timeseries, and sequence data in general. The two fundamental deep-learning algorithms for sequence processing are recurrent neural networks and 1D convnets, the one-dimensional version of the 2D convnets that we covered in the previous chapters. We’ll discuss both of these approaches in this chapter.

Applications of these algorithms include the following: