book

Math and Architectures of Deep Learning

by Krishnendu Chaudhury

May 2024

Intermediate to advanced

552 pages

18h 3m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorsabout the cover illustration
1.1 A first look at machine/deep learning: A paradigm shift in computation1.2 A function approximation view of machine learning:Models and their training1.3 A simple machine learning model: The cat brain1.3.1 Input features1.3.2 Output decisions1.3.3 Model estimation1.3.4 Model architecture selection1.3.5 Model training1.3.6 Inferencing1.4 Geometrical view of machine learning1.5 Regression vs. classification in machine learning1.6 Linear vs. nonlinear models1.7 Higher expressive power through multiple nonlinear layers: Deep neural networksSummary
2.1 Vectors and their role in machine learning2.1.1 The geometric view of vectors and its significance in machine learning2.2 PyTorch code for vector manipulations2.2.1 PyTorch code for the introduction to vectors2.3 Matrices and their role in machine learning2.3.1 Matrix representation of digital images2.4 Python code: Introducing matrices, tensors, and images via PyTorch2.5 Basic vector and matrix operations in machine learning2.5.1 Matrix and vector transpose2.5.2 Dot product of two vectors and its role in machine learning2.5.3 Matrix multiplication and machine learning2.5.4 Length of a vector (L2 norm): Model error2.5.5 Geometric intuitions for vector length2.5.6 Geometric intuitions for the dot product: Feature similarity2.6 Orthogonality of vectors and its physical significance2.7 Python code: Basic vector and matrix operations via PyTorch2.7.1 PyTorch code for a matrix transpose2.7.2 PyTorch code for a dot product2.7.3 PyTorch code for matrix vector multiplication2.7.4 PyTorch code for matrix-matrix multiplication2.7.5 PyTorch code for the transpose of a matrix product2.8 Multidimensional line and plane equations and machine learning2.8.1 Multidimensional line equation2.8.2 Multidimensional planes and their role in machine learning2.9 Linear combinations, vector spans, basis vectors, and collinearity preservation2.9.1 Linear dependence2.9.2 Span of a set of vectors2.9.3 Vector spaces, basis vectors, and closure2.10 Linear transforms: Geometric and algebraic interpretations2.10.1 Generic multidimensional definition of linear transforms2.10.2 All matrix-vector multiplications are linear transforms2.11 Multidimensional arrays, multilinear transforms, and tensors2.11.1 Array view: Multidimensional arrays of numbers2.12 Linear systems and matrix inverse2.12.1 Linear systems with zero or near-zero determinants,and ill-conditioned systems2.12.2 PyTorch code for inverse, determinant, and singularity testing of matrices2.12.3 Over- and under-determined linear systems in machine learning2.12.4 Moore Penrose pseudo-inverse of a matrix2.12.5 Pseudo-inverse of a matrix: A beautiful geometric intuition2.12.6 PyTorch code to solve overdetermined systems2.13 Eigenvalues and eigenvectors: Swiss Army knives of machine learning2.13.1 Eigenvectors and linear independence2.13.2 Symmetric matrices and orthogonal eigenvectors2.13.3 PyTorch code to compute eigenvectors and eigenvalues2.14 Orthogonal (rotation) matrices and their eigenvalues and eigenvectors2.14.1 Rotation matrices2.14.2 Orthogonality of rotation matrices2.14.3 PyTorch code for orthogonality of rotation matrices2.14.4 Eigenvalues and eigenvectors of a rotation matrix:Finding the axis of rotation2.14.5 PyTorch code for eigenvalues and vectors of rotation matrices2.15 Matrix diagonalization2.15.1 PyTorch code for matrix diagonalization2.15.2 Solving linear systems without inversion via diagonalization2.15.3 PyTorch code for solving linear systems via diagonalization2.15.4 Matrix powers using diagonalization2.16 Spectral decomposition of a symmetric matrix2.16.1 PyTorch code for the spectral decomposition of a matrix2.17 An application relevant to machine learning: Finding the axes of a hyperellipse2.17.1 PyTorch code for hyperellipsesSummary
3.1 Geometrical view of image classification3.1.1 Input representation3.1.2 Classifiers as decision boundaries3.1.3 Modeling in a nutshell3.1.4 Sign of the surface function in binary classification3.2 Error, aka loss function3.3 Minimizing loss functions: Gradient vectors3.3.1 Gradients: A machine learning-centric introduction3.3.2 Level surface representation and loss minimization3.4 Local approximation for the loss function3.4.1 1D Taylor series recap3.4.2 Multidimensional Taylor series and the Hessian matrix3.5 PyTorch code for gradient descent, error minimization,and model training3.5.1 PyTorch code for linear models3.5.2 Autograd: PyTorch automatic gradient computation3.5.3 Nonlinear Models in PyTorch3.5.4 A linear model for the cat brain in PyTorch3.6 Convex and nonconvex functions, and global and local minima3.7 Convex sets and functions3.7.1 Convex sets3.7.2 Convex curves and surfaces3.7.3 Convexity and the Taylor series3.7.4 Examples of convex functionsSummary
4.1 Distribution of feature data points and true dimensionality4.2 Quadratic forms and their minimization4.2.1 Minimizing quadratic forms4.2.2 Symmetric positive (semi)definite matrices4.3 Spectral and Frobenius norms of a matrix4.3.1 Spectral norms4.3.2 Frobenius norms4.4 Principal component analysis4.4.1 Direction of maximum spread4.4.2 PCA and dimensionality reduction4.4.3 PyTorch code: PCA and dimensionality reduction4.4.4 Limitations of PCA4.4.5 PCA and data compression4.5 Singular value decomposition4.5.1 Informal proof of the SVD theorem4.5.2 Proof of the SVD theorem4.5.3 Applying SVD: PCA computation4.5.4 Applying SVD: Solving arbitrary linear systems4.5.5 Rank of a matrix4.5.6 PyTorch code for solving linear systems with SVD4.5.7 PyTorch code for PCA computation via SVD4.5.8 Applying SVD: Best low-rank approximation of a matrix4.6 Machine learning application: Document retrieval4.6.1 Using TF-IDF and cosine similarity4.6.2 Latent semantic analysis4.6.3 PyTorch code to perform LSA4.6.4 PyTorch code to compute LSA and SVD on a large datasetSummary
5.1 Probability: The classical frequentist view5.1.1 Random variables5.1.2 Population histograms5.2 Probability distributions5.3 Basic concepts of probability theory5.3.1 Probabilities of impossible and certain events5.3.2 Exhaustive and mutually exclusive events5.3.3 Independent events5.4 Joint probabilities and their distributions5.4.1 Marginal probabilities5.4.2 Dependent events and their joint probability distribution5.5 Geometrical view: Sample point distributions for dependent and independent variables5.6 Continuous random variables and probability density5.7 Properties of distributions: Expected value, variance, and covariance5.7.1 Expected value (aka mean)5.7.2 Variance, covariance, and standard deviation5.8 Sampling from a distribution5.9 Some famous probability distributions5.9.1 Uniform random distributions5.9.2 Gaussian (normal) distribution5.9.3 Binomial distribution5.9.4 Multinomial distribution5.9.5 Bernoulli distribution5.9.6 Categorical distribution and one-hot vectorsSummary
6.1 Conditional probability and Bayes’ theorem6.1.1 Joint and marginal probability revisited6.1.2 Conditional probability6.1.3 Bayes’ theorem6.2 Entropy6.2.1 Geometrical intuition for entropy6.2.2 Entropy of Gaussians6.3 Cross-entropy6.4 KL divergence6.4.1 KLD between Gaussians6.5 Conditional entropy6.5.1 Chain rule of conditional entropy6.6 Model parameter estimation6.6.1 Likelihood, evidence, and posterior and prior probabilities6.6.2 Maximum likelihood parameter estimation (MLE)6.6.3 Maximum a posteriori (MAP) parameter estimation and regularization6.7 Latent variables and evidence maximization6.8 Maximum likelihood parameter estimation for Gaussians6.8.1 Python PyTorch code for maximum likelihood estimation6.8.2 Python PyTorch code for maximum likelihood estimation using gradient descent6.9 Gaussian mixture models6.9.1 Probability density function of the GMM6.9.2 Latent variables for class selection6.9.3 Classification via GMM6.9.4 Maximum likelihood estimation of GMM parameters (GMM fit)Summary

7.1 Neural networks: A 10,000-foot view7.2 Expressing real-world problems: Target functions7.2.1 Logical functions in real-world problems7.2.2 Classifier functions in real-world problems7.2.3 General functions in real-world problems7.3 The basic building block or neuron: The perceptron7.3.1 The Heaviside step function7.3.2 Hyperplanes7.3.3 Perceptrons and classification7.3.4 Modeling common logic gates with perceptrons7.4 Toward more expressive power: Multilayer perceptrons (MLPs)7.4.1 MLP for logical XOR7.5 Layered networks of perceptrons: MLPs or neural networks7.5.1 Layering7.5.2 Modeling logical functions with MLPs7.5.3 Cybenko’s universal approximation theorem7.5.4 MLPs for polygonal decision boundariesSummary
8.1 Differentiable step-like functions8.1.1 Sigmoid function8.1.2 Tanh function8.2 Why layering?8.3 Linear layers8.3.1 Linear layers expressed as matrix-vector multiplication8.3.2 Forward propagation and grand output functions for an MLP of linear layers8.4 Training and backpropagation8.4.1 Loss and its minimization: Goal of training8.4.2 Loss surface and gradient descent8.4.3 Why a gradient provides the best direction for descent8.4.4 Gradient descent and local minima8.4.5 The backpropagation algorithm8.4.6 Putting it all together: Overall training algorithm8.5 Training a neural network in PyTorchSummary
9.1 Loss functions9.1.1 Quantification and geometrical view of loss9.1.2 Regression loss9.1.3 Cross-entropy loss9.1.4 Binary cross-entropy loss for image and vector mismatches9.1.5 Softmax9.1.6 Softmax cross-entropy loss9.1.7 Focal loss9.1.8 Hinge loss9.2 Optimization9.2.1 Geometrical view of optimization9.2.2 Stochastic gradient descent and minibatches9.2.3 PyTorch code for SGD9.2.4 Momentum9.2.5 Geometric view: Constant loss contours, gradient descent, and momentum9.2.6 Nesterov accelerated gradients9.2.7 AdaGrad9.2.8 Root-mean-squared propagation9.2.9 Adam optimizer9.3 Regularization9.3.1 Minimum descriptor length: An Occam’s razor view of optimization9.3.2 L2 regularization9.3.3 L1 regularization9.3.4 Sparsity: L1 vs. L2 regularization9.3.5 Bayes’ theorem and the stochastic view of optimization9.3.6 DropoutSummary
10.1 One-dimensional convolution: Graphical and algebraical view10.1.1 Curve smoothing via 1D convolution10.1.2 Curve edge detection via 1D convolution10.1.3 One-dimensional convolution as matrix multiplication10.1.4 PyTorch- One-dimensional convolution with custom weights10.2 Convolution output size10.3 Two-dimensional convolution: Graphical and algebraic view10.3.1 Image smoothing via 2D convolution10.3.2 Image edge detection via 2D convolution10.3.3 PyTorch- 2D convolution with custom weights10.3.4 Two-dimensional convolution as matrix multiplication10.4 Three-dimensional convolution10.4.1 Video motion detection via 3D convolution10.4.2 PyTorch- Three-dimensional convolution with custom weights10.5 Transposed convolution or fractionally strided convolution10.5.1 Application of transposed convolution: Autoencoders and embeddings10.5.2 Transposed convolution output size10.5.3 Upsampling via transpose convolution10.6 Adding convolution layers to a neural network10.6.1 PyTorch- Adding convolution layers to a neural network10.7 PoolingSummary
11.1 CNNs for image classification: LeNet11.1.1 PyTorch- Implementing LeNet for image classification on MNIST11.2 Toward deeper neural networks11.2.1 VGG (Visual Geometry Group) Net11.2.2 Inception: Network-in-network paradigm11.2.3 ResNet: Why stacking layers to add depth does not scale11.2.4 PyTorch Lightning11.3 Object detection: A brief history11.3.1 R-CNN11.3.2 Fast R-CNN11.3.3 Faster R-CNN11.4 Faster R-CNN: A deep dive11.4.1 Convolutional backbone11.4.2 Region proposal network11.4.3 Fast R-CNN11.4.4 Training the Faster R-CNN11.4.5 Other object-detection paradigmsSummary
12.1 Manifolds12.1.1 Hausdorff property12.1.2 Second countable property12.2 Homeomorphism12.3 Neural networks and homeomorphism between manifoldsSummary
13.1 Fully Bayes estimation: An informal introduction13.1.1 Parameter estimation and belief injection13.2 MLE for Gaussian parameter values recap)13.3 Fully Bayes parameter estimation: Gaussian, unknown mean, known precision13.4 Small and large volumes of training data, and strong and weak priors13.5 Conjugate priors13.6 Fully Bayes parameter estimation: Gaussian, unknown precision, known mean13.6.1 Estimating the precision parameter13.7 Fully Bayes parameter estimation: Gaussian, unknown mean, unknown precision13.7.1 Normal-gamma distribution13.7.2 Estimating the mean and precision parameters13.8 Example: Fully Bayesian inferencing13.8.1 Maximum likelihood estimation13.8.2 Bayesian inference13.9 Fully Bayes parameter estimation: Multivariate Gaussian, unknown mean, known precision13.10 Fully Bayes parameter estimation: Multivariate, unknown precision, known mean13.10.1 Wishart distribution13.10.2 Estimating precisionSummary
14.1 Geometric view of latent spaces14.2 Generative classifiers14.3 Benefits and applications of latent-space modeling14.4 Linear latent space manifolds and PCA14.4.1 PyTorch code for dimensionality reduction using PCA14.5 Autoencoders14.5.1 Autoencoders and PCA14.6 Smoothness, continuity, and regularization of latent spaces14.7 Variational autoencoders14.7.1 Geometric overview of VAEs14.7.2 VAE training, losses, and inferencing14.7.3 VAEs and Bayes’ theorem14.7.4 Stochastic mapping leads to latent-space smoothness14.7.5 Direct minimization of the posterior requires prohibitively expensive normalization14.7.6 ELBO and VAEs14.7.7 Choice of prior: Zero-mean, unit-covariance Gaussian14.7.8 Reparameterization trickSummary
A.1 Dot product and cosine of the angle between two vectorsA.2 DeterminantsA.3 Computing the variance of a Gaussian distributionA.4 Two theorems in statisticsA.4.1 Jensen’s InequalityA.4.2 Log sum inequalityA.5 Gamma functions and distributionA.5.1 Gamma functionA.5.2 Gamma distribution

Content preview from Math and Architectures of Deep Learning

9 Loss, optimization, and regularization

This chapter covers

Geometrical and algebraic introductions to loss functions
Geometrical intuitions for softmax
Optimization techniques including momentum, Nesterov, AdaGrad, Adam, and SGD
Regularization and its relationship to Bayesian approaches
Overfitting while training, and dropout

By now, it should be etched in your mind that neural networks are essentially function approximators. In particular, neural network classifiers model the decision boundaries between the classes in the feature space (a space where every input feature combination is a specific point). Supervised classifiers mark sample training data inputs in this space with a—perhaps manually generated—class label (ground truth). The ...