Hands-On Deep Learning Algorithms with Python

Hands-On Deep Learning Algorithms with Python

by Sudharsan Ravichandiran
Released July 2019
Publisher(s): Packt Publishing
ISBN: 9781789344158

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Understand basic-to-advanced deep learning algorithms, the mathematical principles behind them, and their practical applications

Key Features

  • Get up to speed with building your own neural networks from scratch
  • Gain insights into the mathematical principles behind deep learning algorithms
  • Implement popular deep learning algorithms such as CNNs, RNNs, and more using TensorFlow

Book Description

Deep learning is one of the most popular domains in the AI space that allows you to develop multi-layered models of varying complexities.

This book introduces you to popular deep learning algorithms—from basic to advanced—and shows you how to implement them from scratch using TensorFlow. Throughout the book, you will gain insights into each algorithm, the mathematical principles involved, and how to implement it in the best possible manner. The book starts by explaining how you can build your own neural networks, followed by introducing you to TensorFlow, the powerful Python-based library for machine learning and deep learning. Moving on, you will get up to speed with gradient descent variants, such as NAG, AMSGrad, AdaDelta, Adam, and Nadam. The book will then provide you with insights into recurrent neural networks (RNNs) and LSTM and how to generate song lyrics with RNN. Next, you will master the math necessary to work with convolutional and capsule networks, widely used for image recognition tasks. You will also learn how machines understand the semantics of words and documents using CBOW, skip-gram, and PV-DM. Finally, you will explore GANs, including InfoGAN and LSGAN, and autoencoders, such as contractive autoencoders and VAE.

By the end of this book, you will be equipped with all the skills you need to implement deep learning in your own projects.

What you will learn

  • Implement basic-to-advanced deep learning algorithms
  • Master the mathematics behind deep learning algorithms
  • Become familiar with gradient descent and its variants, such as AMSGrad, AdaDelta, Adam, and Nadam
  • Implement recurrent networks, such as RNN, LSTM, GRU, and seq2seq models
  • Understand how machines interpret images using CNN and capsule networks
  • Implement different types of generative adversarial network, such as CGAN, CycleGAN, and StackGAN
  • Explore various types of autoencoder, such as Sparse autoencoders, DAE, CAE, and VAE

Who this book is for

If you are a machine learning engineer, data scientist, AI developer, or anyone looking to delve into neural networks and deep learning, this book is for you. Those who are completely new to deep learning, but have some experience in machine learning and Python programming will also find the book very helpful.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Hands-On Deep Learning Algorithms with Python
  3. Dedication
  4. About Packt
    1. Why subscribe?
  5. Contributors
    1. About the author
    2. About the reviewers
    3. Packt is searching for authors like you
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Download the color images
      3. Conventions used
    4. Get in touch
      1. Reviews
  7. Section 1: Getting Started with Deep Learning
  8. Introduction to Deep Learning
    1. What is deep learning?
    2. Biological and artificial neurons
    3. ANN and its layers
      1. Input layer
      2. Hidden layer
      3. Output layer
    4. Exploring activation functions
      1. The sigmoid function
      2. The tanh function
      3. The Rectified Linear Unit function
      4. The leaky ReLU function
      5. The Exponential linear unit function
      6. The Swish function
      7. The softmax function
    5. Forward propagation in ANN
    6. How does ANN learn?
    7. Debugging gradient descent with gradient checking
    8. Putting it all together
      1. Building a neural network from scratch
    9. Summary
    10. Questions
    11. Further reading
  9. Getting to Know TensorFlow
    1. What is TensorFlow?
    2. Understanding computational graphs and sessions
      1. Sessions
    3. Variables, constants, and placeholders
      1. Variables
      2. Constants
      3. Placeholders and feed dictionaries
    4. Introducing TensorBoard
      1. Creating a name scope
    5. Handwritten digit classification using TensorFlow
      1. Importing the required libraries
      2. Loading the dataset
      3. Defining the number of neurons in each layer
      4. Defining placeholders
      5. Forward propagation
      6. Computing loss and backpropagation
      7. Computing accuracy
      8. Creating summary
      9. Training the model
      10. Visualizing graphs in TensorBoard
    6. Introducing eager execution
    7. Math operations in TensorFlow
    8. TensorFlow 2.0 and Keras
      1. Bonjour Keras
        1. Defining the model
          1. Defining a sequential model
          2. Defining a functional model
        2. Compiling the model
        3. Training the model
        4. Evaluating the model
      2. MNIST digit classification using TensorFlow 2.0
    9. Should we use Keras or TensorFlow?
    10. Summary
    11. Questions
    12. Further reading
  10. Section 2: Fundamental Deep Learning Algorithms
  11. Gradient Descent and Its Variants
    1. Demystifying gradient descent
      1. Performing gradient descent in regression
        1. Importing the libraries
        2. Preparing the dataset
        3. Defining the loss function
        4. Computing the gradients of the loss function
        5. Updating the model parameters
    2. Gradient descent versus stochastic gradient descent
    3. Momentum-based gradient descent
      1. Gradient descent with momentum
      2. Nesterov accelerated gradient
    4. Adaptive methods of gradient descent
      1. Setting a learning rate adaptively using Adagrad
      2. Doing away with the learning rate using Adadelta
      3. Overcoming the limitations of Adagrad using RMSProp
      4. Adaptive moment estimation
        1. Adamax – Adam based on infinity-norm
      5. Adaptive moment estimation with AMSGrad
      6. Nadam – adding NAG to ADAM
    5. Summary
    6. Questions
    7. Further reading
  12. Generating Song Lyrics Using RNN
    1. Introducing RNNs
      1. The difference between feedforward networks and RNNs
      2. Forward propagation in RNNs
      3. Backpropagating through time
        1. Gradients with respect to the hidden to output weight, V
        2. Gradients with respect to hidden to hidden layer weights, W
        3. Gradients with respect to input to the hidden layer weight, U
      4. Vanishing and exploding gradients problem
        1. Gradient clipping
    2. Generating song lyrics using RNNs
      1. Implementing in TensorFlow
        1. Data preparation
        2. Defining the network parameters
        3. Defining placeholders
        4. Defining forward propagation
        5. Defining BPTT
        6. Start generating songs
    3. Different types of RNN architectures
      1. One-to-one architecture
      2. One-to-many architecture
      3. Many-to-one architecture
      4. Many-to-many architecture
    4. Summary
    5. Questions
    6. Further reading
  13. Improvements to the RNN
    1. LSTM to the rescue
      1. Understanding the LSTM cell
        1. Forget gate
        2. Input gate
        3. Output gate
        4. Updating the cell state
        5. Updating hidden state
      2. Forward propagation in LSTM
      3. Backpropagation in LSTM
        1. Gradients with respect to gates
        2. Gradients with respect to weights
          1. Gradients with respect to V
          2. Gradients with respect to W
          3. Gradients with respect to U
      4. Predicting Bitcoin prices using LSTM model
        1. Data preparation
        2. Defining the parameters
        3. Define the LSTM cell
        4. Defining forward propagation
        5. Defining backpropagation
        6. Training the LSTM model
        7. Making predictions using the LSTM model
    2. Gated recurrent units
      1. Understanding the GRU cell
        1. Update gate
        2. Reset gate
        3. Updating hidden state
      2. Forward propagation in a GRU cell
      3. Backpropagation in a GRU cell
        1. Gradient with respect to gates
        2. Gradients with respect to weights
          1. Gradients with respect to V
          2. Gradients with respect to W
          3. Gradients with respect to U
      4. Implementing a GRU cell in TensorFlow
        1. Defining the weights
        2. Defining forward propagation
    3. Bidirectional RNN
    4. Going deep with deep RNN
    5. Language translation using the seq2seq model
      1. Encoder
      2. Decoder
      3. Attention is all we need
    6. Summary
    7. Questions
    8. Further reading
  14. Demystifying Convolutional Networks
    1. What are CNNs?
      1. Convolutional layers
        1. Strides
        2. Padding
      2. Pooling layers
      3. Fully connected layers
    2. The architecture of CNNs
    3. The math behind CNNs
      1. Forward propagation
      2. Backward propagation
    4. Implementing a CNN in TensorFlow
      1. Defining helper functions
      2. Defining the convolutional network
      3. Computing loss
      4. Starting the training
      5. Visualizing extracted features
    5. CNN architectures
      1. LeNet architecture
      2. Understanding AlexNet
      3. Architecture of VGGNet
      4. GoogleNet
        1. Inception v1
        2. Inception v2 and v3
    6. Capsule networks
      1. Understanding Capsule networks
        1. Computing prediction vectors
        2. Coupling coefficients
        3. Squashing function
      2. Dynamic routing algorithm
      3. Architecture of the Capsule network
      4. The loss function
        1. Margin loss
        2. Reconstruction loss
    7. Building Capsule networks in TensorFlow
      1. Defining the squash function
      2. Defining a dynamic routing algorithm
      3. Computing primary and digit capsules
      4. Masking the digit capsule
      5. Defining the decoder
      6. Computing the accuracy of the model
      7. Calculating loss
        1. Margin loss
        2. Reconstruction loss
        3. Total loss
      8. Training the Capsule network
    8. Summary
    9. Questions
    10. Further reading
  15. Learning Text Representations
    1. Understanding the word2vec model
      1. Understanding the CBOW model
        1. CBOW with a single context word
          1. Forward propagation
          2. Backward propagation
        2. CBOW with multiple context words
      2. Understanding skip-gram model
        1. Forward propagation in skip-gram
        2. Backward propagation
      3. Various training strategies
        1. Hierarchical softmax
        2. Negative sampling
        3. Subsampling frequent words
    2. Building the word2vec model using gensim
      1. Loading the dataset
      2. Preprocessing and preparing the dataset
      3. Building the model
      4. Evaluating the embeddings
    3. Visualizing word embeddings in TensorBoard
    4. Doc2vec
      1. Paragraph Vector – Distributed Memory model
      2. Paragraph Vector – Distributed Bag of Words model
      3. Finding similar documents using doc2vec
    5. Understanding skip-thoughts algorithm
    6. Quick-thoughts for sentence embeddings
    7. Summary
    8. Questions
    9. Further reading
  16. Section 3: Advanced Deep Learning Algorithms
  17. Generating Images Using GANs
    1. Differences between discriminative and generative models
    2. Say hello to GANs!
      1. Breaking down the generator
      2. Breaking down the discriminator
      3. How do they learn though?
      4. Architecture of a GAN
      5. Demystifying the loss function
        1. Discriminator loss
          1. First term
          2. Second term
          3. Final term
        2. Generator loss
        3. Total loss
        4. Heuristic loss
      6. Generating images using GANs in TensorFlow
        1. Reading the dataset
        2. Defining the generator
        3. Defining the discriminator
        4. Defining the input placeholders
        5. Starting the GAN!
        6. Computing the loss function
          1. Discriminator loss
          2. Generator loss
        7. Optimizing the loss
        8. Starting the training
        9. Generating handwritten digits
    3. DCGAN – Adding convolution to a GAN
      1. Deconvolutional generator
      2. Convolutional discriminator
      3. Implementing DCGAN to generate CIFAR images
        1. Exploring the dataset
        2. Defining the discriminator
        3. Defining the generator
        4. Defining the inputs
        5. Starting the DCGAN
        6. Computing the loss function
          1. Discriminator loss
          2. Generator loss
        7. Optimizing the loss
        8. Train the DCGAN
    4. Least squares GAN
      1. Loss function
      2. LSGAN in TensorFlow
        1. Discriminator loss
        2. Generator loss
    5. GANs with Wasserstein distance
      1. Are we minimizing JS divergence in GANs?
      2. What is the Wasserstein distance?
      3. Demystifying the k-Lipschitz function
      4. The loss function of WGAN
      5. WGAN in TensorFlow
    6. Summary
    7. Questions
    8. Further reading
  18. Learning More about GANs
    1. Conditional GANs
      1. Loss function of CGAN
      2. Generating specific handwritten digits using CGAN
        1. Defining the generator
        2. Defining discriminator
        3. Start the GAN!
        4. Computing the loss function
          1. Discriminator loss
          2. Generator loss
        5. Optimizing the loss
        6. Start training the CGAN
          1. Generate the handwritten digit, 7
    2. Understanding InfoGAN
      1. Mutual information
      2. Architecture of the InfoGAN
      3. Constructing an InfoGAN in TensorFlow
        1. Defining generator
        2. Defining the discriminator
        3. Define the input placeholders
        4. Start the GAN
        5. Computing loss function
          1. Discriminator loss
          2. Generator loss
          3. Mutual information
        6. Optimizing the loss
        7. Beginning training
          1. Generating handwritten digits
    3. Translating images using a CycleGAN
      1. Role of generators
      2. Role of discriminators
      3. Loss function
      4. Cycle consistency loss
      5. Converting photos to paintings using a CycleGAN
    4. StackGAN
      1. The architecture of StackGANs
        1. Conditioning augmentation
        2. Stage I
          1. Generator
          2. Discriminator
        3. Stage II
          1. Generator
          2. Discriminator
    5. Summary
    6. Questions
    7. Further reading
  19. Reconstructing Inputs Using Autoencoders
    1. What is an autoencoder?
      1. Understanding the architecture of autoencoders
      2. Reconstructing the MNIST images using an autoencoder
        1. Preparing the dataset
        2. Defining the encoder
        3. Defining the decoder
        4. Building the model
        5. Reconstructing images
        6. Plotting reconstructed images
    2. Autoencoders with convolutions
      1. Building a convolutional autoencoder
        1. Defining the encoder
        2. Defining the decoder
        3. Building the model
        4. Reconstructing the images
    3. Exploring denoising autoencoders
      1. Denoising images using DAE
    4. Understanding sparse autoencoders
      1. Building the sparse autoencoder
        1. Defining the sparse regularizer
    5. Learning to use contractive autoencoders
      1. Implementing the contractive autoencoder
        1. Defining the contractive loss
    6. Dissecting variational autoencoders
      1. Variational inference
      2. The loss function
        1. Reparameterization trick
      3. Generating images using VAE
        1. Preparing the dataset
        2. Defining the encoder
        3. Defining the sampling operation
        4. Defining the decoder
        5. Building the model
        6. Defining the generator
        7. Plotting generated images
    7. Summary
    8. Questions
    9. Further reading
  20. Exploring Few-Shot Learning Algorithms
    1. What is few-shot learning?
    2. Siamese networks
    3. Architecture of siamese networks
    4. Prototypical networks
    5. Relation networks
    6. Matching networks
      1. Support set embedding function
      2. Query set embedding function
      3. The architecture of matching networks
    7. Summary
    8. Questions
    9. Further reading
  21. Assessments
    1. Chapter 1 - Introduction to Deep Learning
    2. Chapter 2 - Getting to Know TensorFlow
    3. Chapter 3 - Gradient Descent and Its Variants
    4. Chapter 4 - Generating Song Lyrics Using an RNN
    5. Chapter 5 - Improvements to the RNN
    6. Chapter 6 - Demystifying Convolutional Networks
    7. Chapter 7 - Learning Text Representations
    8. Chapter 8 - Generating Images Using GANs
    9. Chapter 9 - Learning More about GANs
    10. Chapter 10 - Reconstructing Inputs Using Autoencoders
    11. Chapter 11 - Exploring Few-Shot Learning Algorithms
  22. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Hands-On Deep Learning Algorithms with Python
  • Author(s): Sudharsan Ravichandiran
  • Release date: July 2019
  • Publisher(s): Packt Publishing
  • ISBN: 9781789344158