Deep Learning Essentials

Table of contents

1. Preface
   1. Who this book is for
   2. What this book covers
   3. To get the most out of this book
      1. Download the example code files
      2. Download the color images
      3. Conventions used
   4. Get in touch
      1. Reviews
2. Why Deep Learning?
   1. What is AI and deep learning?
   2. The history and rise of deep learning
   3. Why deep learning?
      1. Advantages over traditional shallow methods
      2. Impact of deep learning
   4. The motivation of deep architecture
      1. The neural viewpoint
      2. The representation viewpoint
         1. Distributed feature representation
         2. Hierarchical feature representation
   5. Applications
      1. Lucrative applications
      2. Success stories
      3. Deep learning for business
   6. Future potential and challenges
   7. Summary
3. Getting Yourself Ready for Deep Learning
   1. Basics of linear algebra
      1. Data representation
      2. Data operations
      3. Matrix properties
   2. Deep learning with GPU
      1. Deep learning hardware guide
         1. CPU cores
         2. CPU cache size
         3. RAM size
         4. Hard drive
         5. Cooling systems
   3. Deep learning software frameworks
      1. TensorFlow – a deep learning library
      2. Caffe
      3. MXNet
      4. Torch
      5. Theano
      6. Microsoft Cognitive Toolkit
      7. Keras
      8. Framework comparison
   4. Setting up deep learning on AWS
      1. Setup from scratch
      2. Setup using Docker
   5. Summary
4. Getting Started with Neural Networks
   1. Multilayer perceptrons
      1. The input layer
      2. The output layer
      3. Hidden layers
      4. Activation functions
         1. Sigmoid or logistic function
         2. Tanh or hyperbolic tangent function
         3. ReLU
         4. Leaky ReLU and maxout
         5. Softmax
         6. Choosing the right activation function
   2. How a network learns
      1. Weight initialization
      2. Forward propagation
      3. Backpropagation
         1. Calculating errors
         2. Backpropagation
         3. Updating the network
         4. Automatic differentiation
      4. Vanishing and exploding gradients
      5. Optimization algorithms
      6. Regularization
   3. Deep learning models
      1. Convolutional Neural Networks
         1. Convolution
         2. Pooling/subsampling
         3. Fully connected layer
         4. Overall
      2. Restricted Boltzmann Machines
         1. Energy function
         2. Encoding and decoding
         3. Contrastive divergence (CD-k)
         4. Stacked/continuous RBM
         5. RBM versus Boltzmann Machines
      3. Recurrent neural networks (RNN/LSTM)
         1. Cells in RNN and unrolling
         2. Backpropagation through time
         3. Vanishing gradient and LTSM
         4. Cells and gates in LTSM
            1. Step 1 – The forget gate
            2. Step 2 – Updating memory/cell state
            3. Step 3 – The output gate
   4. Practical examples
      1. TensorFlow setup and key concepts
      2. Handwritten digits recognition
   5. Summary
5. Deep Learning in Computer Vision
   1. Origins of CNNs
   2. Convolutional Neural Networks
      1. Data transformations
         1. Input preprocessing
         2. Data augmentation
      2. Network layers
         1. Convolution layer
         2. Pooling or subsampling layer
         3. Fully connected or dense layer
      3. Network initialization
      4. Regularization
      5. Loss functions
      6. Model visualization
      7. Handwritten digit classification example
   3. Fine-tuning CNNs
   4. Popular CNN architectures
      1. AlexNet
      2. Visual Geometry Group
      3. GoogLeNet
      4. ResNet
   5. Summary
6. NLP - Vector Representation
   1. Traditional NLP
      1. Bag of words
      2. Weighting the terms tf-idf
   2. Deep learning NLP
      1. Motivation and distributed representation
      2. Word embeddings
         1. Idea of word embeddings
         2. Advantages of distributed representation
         3. Problems of distributed representation
         4. Commonly used pre-trained word embeddings
      3. Word2Vec
         1. Basic idea of Word2Vec
         2. The word windows
         3. Generating training data
         4. Negative sampling
         5. Hierarchical softmax
         6. Other hyperparameters
         7. Skip-Gram model
            1. The input layer
            2. The hidden layer
            3. The output layer
            4. The loss function
         8. Continuous Bag-of-Words model
         9. Training a Word2Vec using TensorFlow
         10. Using existing pre-trained Word2Vec embeddings
             1. Word2Vec from Google News
             2. Using the pre-trained Word2Vec embeddings
      4. Understanding GloVe
      5. FastText
   3. Applications
      1. Example use cases
      2. Fine-tuning
   4. Summary
7. Advanced Natural Language Processing
   1. Deep learning for text
      1. Limitations of neural networks
   2. Recurrent neural networks
      1. RNN architectures
      2. Basic RNN model
      3. Training RNN is tough
   3. Long short-term memory network
      1. LSTM implementation with tensorflow
   4. Applications
      1. Language modeling
      2. Sequence tagging
      3. Machine translation
         1. Seq2Seq inference
      4. Chatbots
   5. Summary
8. Multimodality
   1. What is multimodality learning?
   2. Challenges of multimodality learning
      1. Representation
      2. Translation
      3. Alignment
      4. Fusion
      5. Co-learning
   3. Image captioning
      1. Show and tell
         1. Encoder
         2. Decoder
         3. Training
         4. Testing/inference
         5. Beam Search
      2. Other types of approaches
      3. Datasets
      4. Evaluation
         1. BLEU
         2. ROUGE
         3. METEOR
         4. CIDEr
         5. SPICE
         6. Rank position
      5. Attention models
         1. Attention in NLP
         2. Attention in computer vision
            1. The difference between hard attention and soft attention
   4. Visual question answering
   5. Multi-source based self-driving
   6. Summary
9. Deep Reinforcement Learning
   1. What is reinforcement learning (RL)?
      1. Problem setup
      2. Value learning-based algorithms
      3. Policy search-based algorithms
      4. Actor-critic-based algorithms
   2. Deep reinforcement learning
      1. Deep Q-network (DQN)
         1. Experience replay
         2. Target network
         3. Reward clipping
      2. Double-DQN
         1. Prioritized experience delay
      3. Dueling DQN
   3. Implementing reinforcement learning
      1. Simple reinforcement learning example
      2. Reinforcement learning with Q-learning example
   4. Summary
10. Deep Learning Hacks
    1. Massaging your data
       1. Data cleaning
       2. Data augmentation
       3. Data normalization
    2. Tricks in training
       1. Weight initialization
          1. All-zero
          2. Random initialization
          3. ReLU initialization
          4. Xavier initialization
       2. Optimization
          1. Learning rate
          2. Mini-batch
          3. Clip gradients
       3. Choosing the loss function
          1. Multi-class classification
          2. Multi-class multi-label classification
          3. Regression
          4. Others
       4. Preventing overfitting
          1. Batch normalization
          2. Dropout
          3. Early stopping
          4. Fine-tuning
    3. Fine-tuning
       1. When to use fine-tuning
       2. When not to use fine-tuning
       3. Tricks and techniques
       4. List of pre-trained models
    4. Model compression
    5. Summary
11. Deep Learning Trends
    1. Recent models for deep learning
       1. Generative Adversarial Networks
       2. Capsule networks
    2. Novel applications
       1. Genomics
       2. Predictive medicine
       3. Clinical imaging
       4. Lip reading
       5. Visual reasoning
       6. Code synthesis
    3. Summary
12. Other Books You May Enjoy
    1. Leave a review – let other readers know what you think