Hands-On Neural Networks with Keras

Hands-On Neural Networks with Keras

by Niloy Purkait
Released March 2019
Publisher(s): Packt Publishing
ISBN: 9781789536089

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Your one-stop guide to learning and implementing artificial neural networks with Keras effectively

Key Features

  • Design and create neural network architectures on different domains using Keras
  • Integrate neural network models in your applications using this highly practical guide
  • Get ready for the future of neural networks through transfer learning and predicting multi network models

Book Description

Neural networks are used to solve a wide range of problems in different areas of AI and deep learning.

Hands-On Neural Networks with Keras will start with teaching you about the core concepts of neural networks. You will delve into combining different neural network models and work with real-world use cases, including computer vision, natural language understanding, synthetic data generation, and many more. Moving on, you will become well versed with convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, autoencoders, and generative adversarial networks (GANs) using real-world training datasets. We will examine how to use CNNs for image recognition, how to use reinforcement learning agents, and many more. We will dive into the specific architectures of various networks and then implement each of them in a hands-on manner using industry-grade frameworks.

By the end of this book, you will be highly familiar with all prominent deep learning models and frameworks, and the options you have when applying deep learning to real-world scenarios and embedding artificial intelligence as the core fabric of your organization.

What you will learn

  • Understand the fundamental nature and workflow of predictive data modeling
  • Explore how different types of visual and linguistic signals are processed by neural networks
  • Dive into the mathematical and statistical ideas behind how networks learn from data
  • Design and implement various neural networks such as CNNs, LSTMs, and GANs
  • Use different architectures to tackle cognitive tasks and embed intelligence in systems
  • Learn how to generate synthetic data and use augmentation strategies to improve your models
  • Stay on top of the latest academic and commercial developments in the field of AI

Who this book is for

This book is for machine learning practitioners, deep learning researchers and AI enthusiasts who are looking to get well versed with different neural network architecture using Keras. Working knowledge of Python programming language is mandatory.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Hands-On Neural Networks with Keras
  3. About Packt
    1. Why subscribe?
    2. Packt.com
  4. Contributors
    1. About the author
    2. About the reviewer
    3. Packt is searching for authors like you
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Download the color images
      3. Conventions used
    4. Get in touch
      1. Reviews
  6. Section 1: Fundamentals of Neural Networks
  7. Overview of Neural Networks
    1. Defining our goal
    2. Knowing our tools
      1. Keras
      2. TensorFlow
    3. The fundamentals of neural learning
      1. What is a neural network?
      2. Observing the brain
        1. Building a biological brain
        2. The physiology of a neuron
        3. Representing information
        4. The mysteries of neural encoding
        5. Distributed representation and learning
    4. The fundamentals of data science
      1. Information theory
      2. Entropy
      3. Cross entropy
      4. The nature of data processing
      5. From data science to ML
        1. Modeling data in high-dimensional spaces
          1. The curse of dimensionality
          2. Algorithmic computation and predictive models
        2. Matching a model to use cases
        3. Functional representations
      6. The pitfalls of ML
        1. Unbalanced class priors
        2. Underfitting
        3. Overfitting
        4. Bad data
        5. Irrelevant features and labels
    5. Summary
    6. Further reading
  8. A Deeper Dive into Neural Networks
    1. From the biological to the artificial neuron – the perceptron
    2. Building a perceptron
      1. Input
      2. Weights
      3. Summation
      4. Introducing non-linearity
        1. Activation functions
      5. Understanding the role of the bias term
      6. Output
    3. Learning through errors
      1. The mean squared error loss function
    4. Training a perceptron
      1. Quantifying loss
      2. Loss as a function of model weights
    5. Backpropagation
      1. Computing the gradient
      2. The learning rate
    6. Scaling the perceptron
    7. A single layered network
      1. Experimenting with TensorFlow playground
      2. Capturing patterns heirarchically
      3. Steps ahead
    8. Summary
  9. Signal Processing - Data Analysis with Neural Networks
    1. Processing signals
      1. Representational learning
      2. Avoiding random memorization
      3. Representing signals with numbers
    2. Images as numbers
    3. Feeding a neural network
    4. Examples of tensors
      1. Dimensionality of data
      2. Making some imports
        1. Keras's sequential API
      3. Loading the data
      4. Checking the dimensions
    5. Building a model
      1. Introducting Keras layers
      2. Initializing weights
      3. Keras activations
      4. Summarizing your model visually
    6. Compiling the model
      1. Fitting the model
    7. Evaluating model performance
      1. Regularization
        1. Adjusting network size
        2. Size experiments
        3. Regularizing the weights
        4. Using dropout layers
        5. Thinking about dropout intuitively
    8. Implementing weight regularization in Keras
    9. Weight regularization experiments
    10. Implementing dropout regularization in Keras
      1. Dropout regularization experiments
      2. Complexity and time
      3. A summary of MNIST
    11. Language processing
      1. Sentiment analysis
    12. The internet movie reviews dataset
      1. Loading the dataset
      2. Checking the shape and type
    13. Plotting a single training instance
      1. Decoding the reviews
      2. Preparing the data
    14. One-hot encoding
    15. Vectorizing features
    16. Vectorizing labels
    17. Building a network
      1. Compiling the model
      2. Fitting the model
      3. Validation data
    18. Callbacks
      1. Early stopping and history callbacks
      2. Choosing a metric to monitor
    19. Accessing model predictions
    20. Probing the predictions
      1. Summary of IMDB
      2. Predicting continuous variables
        1. Boston Housing Prices dataset
        2. Loading the data
        3. Exploring the data
    21. Feature-wise normalization
      1. Building the model
      2. Compiling the model
      3. Plotting training and test errors
      4. Validating your approach using k-fold validation
    22. Cross validation with scikit-learn API
    23. Summary
    24. Exercises
  10. Section 2: Advanced Neural Network Architectures
  11. Convolutional Neural Networks
    1. Why CNNs?
    2. The birth of vision
    3. Understanding biological vision
    4. Conceptualizing spatial invariance
    5. Defining receptive fields of neurons
    6. Implementing a hierarchy of neurons
    7. The birth of the modern CNN
    8. Designing a CNN
      1. Dense versus convolutional layer
    9. The convolution operation
      1. Preserving the spatial structure of an image
      2. Receptive field
      3. Feature extraction using filters
      4. Backpropagation of errors in CNNs
      5. Using multiple filters
      6. Stride of a convolution
      7. What are features?
    10. Visualizing feature extraction with filters
    11. Looking at complex filters
    12. Summarizing the convolution operation
    13. Understanding pooling layers
      1. Types of pooling operations
    14. Implementing CNNs in Keras
      1. Probing our data
      2. Verifying the data shape
      3. Normalizing our data
      4. Making some imports
    15. Convolutional layer
      1. Defining the number and size of the filters
      2. Padding input tensors
      3. Max Pooling layer
    16. Leveraging a fully connected layer for classification
    17. Summarizing our model
      1. Compiling the model
    18. Checking model accuracy
    19. The problem with detecting smiles
      1. Inside the black box
      2. Neural network fails
      3. Visualizing ConvNet learnings
      4. Visualizing neural activations of intermediate layers
      5. Predictions on an input image
    20. Introducing Keras's functional API
    21. Verifying the number of channels per layer
      1. Visualizing activation maps
    22. Understanding saliency
    23. Visualizing saliency maps with ResNet50
    24. Loading pictures from a local directory
    25. Using Keras's visualization module
    26. Searching through layers
    27. Exercise
    28. Gradient weighted class activation mapping
    29. Visualizing class activations with Keras-vis
    30. Using the pretrained model for prediction
    31. Visualizing maximal activations per output class
    32. Converging a model
    33. Using multiple filter indices to hallucinate
    34. Problems with CNNs
    35. Neural network pareidolia
    36. Summary
  12. Recurrent Neural Networks
    1. Modeling sequences
    2. Using RNNs for sequential modeling
      1. What's the catch?
      2. Basic RNN architecture
        1. Temporarily shared weights
        2. Sequence modeling variations in RNNs
        3. Encoding many-to-many representations
        4. Many-to-one
        5. One-to-many
        6. One-to-many for image captioning
    3. Summarizing different types of sequence processing tasks
      1. How do RNNs learn?
      2. A generic RNN layer
      3. Forward propagation
      4. Computing activations per time step
      5. Simplifying the activation equation
    4. Predicting an output per time step
      1. The problem of unidirectional information flow
      2. The problems of long-term dependencies
    5. Backpropagation through time
      1. Visualizing backpropagation through time
    6. Exploding and vanishing gradients
      1. Thinking on the gradient level
      2. Preventing exploding gradients through clipping
      3. Preventing vanishing gradients with memory
    7. GRUs
      1. The memory cell
        1. Representing the memory cell
        2. Updating the memory value
        3. Mathematics of the update equation
      2. Implementing the no-update scenario
      3. Implementing the update scenario
      4. Preserving relevance between time steps
      5. Formalizing the relevance gate
    8. Building character-level language models in Keras
      1. Loading in Shakespeare's Hamlet
      2. Building a dictionary of characters
      3. Preparing training sequences of characters
      4. Printing out example sequences
      5. Vectorizing the training data
    9. Statistics of character modeling
      1. Modeling character-level probabilities
      2. Sampling thresholds
    10. The purpose of controlling stochasticity
      1. Greedy sampling
      2. Stochastic sampling
    11. Testing different RNN models
      1. Using custom callbacks to generate text
      2. Testing multiple models
    12. Building a SimpleRNN
      1. Stacking RNN layers
    13. Building GRUs
      1. Building bi-directional GRUs
    14. On processing reality sequentially
      1. Benefits of re-ordering sequential data
    15. Bi-directional layer in Keras
      1. Implementing recurrent dropout
    16. Visualizing output values
      1. Visualizing the output of heavier GRU models
    17. Summary
    18. Further reading
    19. Exercise
  13. Long Short-Term Memory Networks
    1. On processing complex sequences
      1. Breaking down memory
    2. The LSTM network
    3. Dissecting the LSTM
      1. Comparing the closest known relative
      2. GRU memory
      3. LSTM memory cell
      4. Treating activations and memory separately
    4. LSTM memory block
      1. Importance of the forget gate
      2. Conceptualizing the difference
      3. Walking through the LSTM
    5. Visualizing the flow of information
      1. Computing cell state
    6. Computing contender memory
    7. Computing activations per timestep
    8. Variations of LSTM and performance
    9. Understanding peephole connections
    10. Importance of timing and counting
      1. Exploring other architectural variations
    11. Putting our knowledge to use
    12. On modeling stock market data
      1. Importing the data
      2. Sorting and visualizing the trend
      3. From DataFrame to tensor
      4. Splitting up the data
      5. Plotting out training and testing splits
      6. Windowed normalization
    13. Denoising the data
    14. Implementing exponential smoothing
      1. Visualizing the curve
      2. Performing one-step-ahead predictions
      3. Simple moving average prediction
      4. Exponential moving average prediction
    15. The problem with one-step-ahead predictions
    16. Creating sequences of observations
      1. Reshaping the data
      2. Making some imports
      3. Baseline neural networks
      4. Building a feedforward network
      5. Recurrent baseline
    17. Building LSTMs
      1. Stacked LSTM
      2. Using helper functions
      3. Training the model
      4. Visualizing results
    18. Closing comments
    19. Summary
    20. Exercises
  14. Reinforcement Learning with Deep Q-Networks
    1. On reward and gratification
      1. A new way of examining learning
    2. Conditioning machines with reinforcement learning
      1. The credit assignment problem
    3. The explore-exploit dilemma
    4. Path to artificial general intelligence
    5. Simulating environments
      1. Understanding states, actions, and rewards
    6. A self-driving taxi cab
      1. Understanding the task
      2. Rendering the environment
      3. Referencing observation space
      4. Referencing action space
      5. Interacting with the environment
      6. Solving the environment randomly
    7. Trade-off between immediate and future rewards
    8. Discounting future rewards
    9. Markov decision process
    10. Understanding policy functions
    11. Assessing the value of a state
    12. Assessing the quality of an action
    13. Using the Bellman equation
    14. Updating the Bellman equation iteratively
    15. Why use neural networks?
    16. Performing a forward pass in Q-learning
    17. Performing a backward pass in Q-Learning
      1. Replacing iterative updates with deep learning
    18. Deep Q-learning in Keras
      1. Making some imports
      2. Preprocessing techniques
      3. Defining input parameters
      4. Making an Atari game state processor
        1. Processing individual states
        2. Processing states in batch
        3. Processing rewards
        4. Limitations of reward clipping
        5. Initializing the environment
      5. Building the network
      6. Absence of pooling layers
      7. Problems with live learning
      8. Storing experience in replay memory
    19. Balancing exploration with exploitation
      1. Epsilon-greedy exploration policy
    20. Initializing the deep Q-learning agent
      1. Training the model
      2. Testing the model
      3. Summarizing the Q-learning algorithm
    21. Double Q-learning
    22. Dueling network architecture
    23. Exercise
      1. Limits of Q-learning
      2. Improving Q-learning with policy gradients
    24. Summary
  15. Section 3: Hybrid Model Architecture
  16. Autoencoders
    1. Why autoencoders?
    2. Automatically encoding information
    3. Understanding the limitations of autoencoders
    4. Breaking down the autoencoder
    5. Training an autoencoder
    6. Overviewing autoencoder archetypes
    7. Network size and representational power
    8. Understanding regularization in autoencoders
    9. Regularization with sparse autoencoders
      1. Regularization with denoising autoencoders
      2. Regularization with contractive autoencoders
      3. Implementing a shallow AE in Keras
        1. Making some imports
    10. Probing the data
      1. Preprocessing the data
      2. Building the model
        1. Implementing a sparsity constraint
        2. Compiling and visualizing the model
    11. Building the verification model
      1. Defining a separate encoder network
      2. Defining a separate decoder network
        1. Training the autoencoder
        2. Visualizing the results
    12. Designing a deep autoencoder
      1. Making some imports
      2. Understanding the data
      3. Importing the data
        1. Preprocessing the data
        2. Partitioning the data
    13. Using functional API to design autoencoders
      1. Building the model
      2. Training the model
      3. Visualizing the results
    14. Deep convolutional autoencoder
    15. Compiling and training the model
    16. Testing and visualizing the results
    17. Denoising autoencoders
    18. Training the denoising network
      1. Visualizing the results
    19. Summary
    20. Exercise
  17. Generative Networks
    1. Replicating versus generating content
    2. Understanding the notion of latent space
      1. Identifying concept vectors
    3. Diving deeper into generative networks
      1. Controlled randomness and creativity
    4. Using randomness to augment outputs
    5. Sampling from the latent space
      1. Learning a probability distribution
    6. Understanding types of generative networks
    7. Understanding VAEs
    8. Designing a VAE in Keras
      1. Loading and pre-processing the data
    9. Building the encoding module in a VAE
      1. Sampling the latent space
    10. Building the decoder module
      1. Defining a custom variational layer
      2. Compiling and inspecting the model
        1. Initiating the training session
    11. Visualizing the latent space
    12. Latent space sampling and output generation
      1. Concluding remarks on VAEs
    13. Exploring GANs
      1. Utility and practical applications for GANS
    14. Diving deeper into GANs
      1. Problems with optimizing GANs
    15. Designing a GAN in Keras
      1. Preparing the data
      2. Visualizing some instances
      3. Pre-processing the data
    16. Designing the generator module
    17. Designing the discriminator module
    18. Putting the GAN together
      1. Helper functions for training
      2. Helper functions to display output
    19. The training function
      1. Arguments in the training function
    20. Defining the discriminator labels
      1. Initializing the GAN
      2. Training the discriminator per batch
    21. Training the generator per batch
      1. Evaluate results per epoch
    22. Executing the training session
      1. Interpreting test loss during training
      2. Visualizing results across epochs
    23. Conclusion
    24. Summary
  18. Section 4: Road Ahead
  19. Contemplating Present and Future Developments
    1. Sharing representations with transfer learning
      1. Transfer learning on Keras
      2. Loading a pretrained model
      3. Obtaining intermediate layers from a model
      4. Adding layers to a model
      5. Loading and preprocessing the data
      6. Training the network
      7. Exercises
    2. Concluding our experiments
    3. Learning representations
      1. DNA and technology
    4. Limits of current neural networks
      1. Engineering representations for machines
        1. How should a good representation be?
      2. Preprocessing and data treatments
        1. Vectorization
        2. Normalization
      3. Smoothness of the data
    5. Encouraging sparse representation learning
    6. Tuning hyperparameters
    7. Automatic optimization and evolutionary algorithms
      1. References
    8. Multi-network predictions and ensemble models
    9. The future of AI and neural networks
      1. Global vectors approach
      2. Distributed representation
      3. Hardware hurdles
    10. The road ahead
    11. Problems with classical computing
    12. The advent of quantum computing
      1. Quantum superposition
      2. Distinguishing Q-Bits from classical counterparts
    13. Quantum neural networks
      1. Further reading
    14. Technology and society
    15. Contemplating our future
    16. Summary 
  20. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Hands-On Neural Networks with Keras
  • Author(s): Niloy Purkait
  • Release date: March 2019
  • Publisher(s): Packt Publishing
  • ISBN: 9781789536089