book

Deep Learning from Scratch

by Seth Weidman

September 2019

Intermediate to advanced

250 pages

6h 58m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Understanding Neural Networks Requires Multiple Mental ModelsChapter OutlinesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
FunctionsMathDiagramsCodeDerivativesMathDiagramsCodeNested FunctionsDiagramMathCodeAnother DiagramThe Chain RuleMathCodeA Slightly Longer ExampleMathDiagramCodeFunctions with Multiple InputsMathDiagramCodeDerivatives of Functions with Multiple InputsDiagramMathCodeFunctions with Multiple Vector InputsMathCreating New Features from Existing FeaturesMathDiagramCodeDerivatives of Functions with Multiple Vector InputsDiagramMathCodeVector Functions and Their Derivatives: One Step FurtherDiagramMathCodeVector Functions and Their Derivatives: The Backward PassComputational Graph with Two 2D Matrix InputsMathDiagramCodeThe Fun Part: The Backward PassDiagramMathCodeConclusion
Supervised Learning OverviewSupervised Learning ModelsLinear RegressionLinear Regression: A DiagramLinear Regression: A More Helpful Diagram (and the Math)Adding in the InterceptLinear Regression: The CodeTraining the ModelCalculating the Gradients: A DiagramCalculating the Gradients: The Math (and Some Code)Calculating the Gradients: The (Full) CodeUsing These Gradients to Train the ModelAssessing Our Model: Training Set Versus Testing SetAssessing Our Model: The CodeAnalyzing the Most Important FeatureNeural Networks from ScratchStep 1: A Bunch of Linear RegressionsStep 2: A Nonlinear FunctionStep 3: Another Linear RegressionDiagramsCodeNeural Networks: The Backward PassTraining and Assessing Our First Neural NetworkTwo Reasons Why This Is HappeningConclusion
Deep Learning Definition: A First PassThe Building Blocks of Neural Networks: OperationsDiagramCodeThe Building Blocks of Neural Networks: LayersDiagramsBuilding Blocks on Building BlocksThe Layer BlueprintThe Dense LayerThe NeuralNetwork Class, and Maybe OthersDiagramCodeLoss ClassDeep Learning from ScratchImplementing Batch TrainingNeuralNetwork: CodeTrainer and OptimizerOptimizerTrainerPutting Everything TogetherOur First Deep Learning Model (from Scratch)Conclusion and Next Steps
Some Intuition About Neural NetworksThe Softmax Cross Entropy Loss FunctionComponent #1: The Softmax FunctionComponent #2: The Cross Entropy LossA Note on Activation FunctionsExperimentsData PreprocessingModelExperiment: Softmax Cross Entropy LossMomentumIntuition for MomentumImplementing Momentum in the Optimizer ClassExperiment: Stochastic Gradient Descent with MomentumLearning Rate DecayTypes of Learning Rate DecayExperiments: Learning Rate DecayWeight InitializationMath and CodeExperiments: Weight InitializationDropoutDefinitionImplementationExperiments: DropoutConclusion
Neural Networks and Representation LearningA Different Architecture for Image DataThe Convolution OperationThe Multichannel Convolution OperationConvolutional LayersImplementation ImplicationsThe Differences Between Convolutional and Fully Connected LayersMaking Predictions with Convolutional Layers: The Flatten LayerPooling LayersImplementing the Multichannel Convolution OperationThe Forward PassConvolutions: The Backward PassBatches, 2D Convolutions, and Multiple Channels2D ConvolutionsThe Last Element: Adding “Channels”Using This Operation to Train a CNNThe Flatten OperationThe Full Conv2D LayerExperimentsConclusion
The Key Limitation: Handling BranchingAutomatic DifferentiationCoding Up Gradient AccumulationMotivation for Recurrent Neural NetworksIntroduction to Recurrent Neural NetworksThe First Class for RNNs: RNNLayerThe Second Class for RNNs: RNNNodePutting These Two Classes TogetherThe Backward PassRNNs: The CodeThe RNNLayer ClassThe Essential Elements of RNNNodes“Vanilla” RNNNodesLimitations of “Vanilla” RNNNodesOne Solution: GRUNodesLSTMNodesData Representation for a Character-Level RNN-Based Language ModelOther Language Modeling TasksCombining RNNLayer VariantsPutting This All TogetherConclusion
PyTorch TensorsDeep Learning with PyTorchPyTorch Elements: Model, Layer, Optimizer, and LossImplementing Neural Network Building Blocks Using PyTorch: DenseLayerExample: Boston Housing Prices Model in PyTorchPyTorch Elements: Optimizer and LossPyTorch Elements: TrainerTricks to Optimize Learning in PyTorchConvolutional Neural Networks in PyTorchDataLoader and TransformsLSTMs in PyTorchPostscript: Unsupervised Learning via AutoencodersRepresentation LearningAn Approach for Situations with No Labels WhatsoeverImplementing an Autoencoder in PyTorchA Stronger Test for Unsupervised Learning, and a SolutionConclusion
Matrix Chain RuleGradient of the Loss with Respect to the Bias TermsConvolutions via Matrix Multiplication

Overview

With the resurgence of neural networks in the 2010s, deep learning has become essential for machine learning practitioners and even many software engineers. This book provides a comprehensive introduction for data scientists and software engineers with machine learning experience. You’ll start with deep learning basics and move quickly to the details of important advanced architectures, implementing everything from scratch along the way.

Author Seth Weidman shows you how neural networks work using a first principles approach. You’ll learn how to apply multilayer neural networks, convolutional neural networks, and recurrent neural networks from the ground up. With a thorough understanding of how neural networks work mathematically, computationally, and conceptually, you’ll be set up for success on all future deep learning projects.

This book provides:

Extremely clear and thorough mental models—accompanied by working code examples and mathematical explanations—for understanding neural networks
Methods for implementing multilayer neural networks from scratch, using an easy-to-understand object-oriented framework
Working implementations and clear-cut explanations of convolutional and recurrent neural networks
Implementation of these neural network concepts using the popular PyTorch framework