book

Mathematics of Machine Learning

by Tivadar Danka

May 2025

Intermediate to advanced

730 pages

20h 14m

English

Packt Publishing

Read now

Unlock full access

What is this book about?How to read this bookConventions usedWhat this book coversTo get the most out of this book
1.1 What is a vector space?1.1.1 Examples of vector spaces1.2 The basis1.2.1 Linear combinations and independence1.2.2 Spans of vector sets1.2.3 Bases, the minimal generating sets1.2.4 Finite dimensional vector spaces1.2.5 Why are bases so important?1.2.6 The existence of bases1.2.7 Subspaces1.3 Vectors in practice1.3.1 Tuples1.3.2 Lists1.3.3 NumPy arrays1.3.4 NumPy arrays as vectors1.3.5 Is NumPy really faster than Python?1.4 Summary1.5 Problems
2.1 Norms and distances2.1.1 Defining distances from norms2.2 Inner products, angles, and lots of reasons to care about them2.2.1 The generated norm2.2.2 Orthogonality2.2.3 The geometric interpretation of inner products2.2.4 Orthogonal and orthonormal bases2.2.5 The Gram-Schmidt orthogonalization process2.2.6 The orthogonal complement2.3 Summary2.4 Problems
3.1 Vectors in NumPy3.1.1 Norms, distances, and dot products3.1.2 The Gram-Schmidt orthogonalization process3.2 Matrices, the workhorses of linear algebra3.2.1 Manipulating matrices3.2.2 Matrices as arrays3.2.3 Matrices in NumPy3.2.4 Matrix multiplication, revisited3.2.5 Matrices and data3.3 Summary3.4 Problems
4.1 What is a linear transformation?4.1.1 Linear transformations and matrices4.1.2 Matrix operations revisited4.1.3 Inverting linear transformations4.1.4 The kernel and the image4.2 Change of basis4.2.1 The transformation matrix4.3 Linear transformations in the Euclidean plane4.3.1 Stretching4.3.2 Rotations4.3.3 Shearing4.3.4 Reflection4.3.5 Orthogonal projection4.4 Determinants, or how linear transformations affect volume4.4.1 How linear transformations scale the area4.4.2 The multi-linearity of determinants4.4.3 Fundamental properties of the determinants4.5 Summary4.6 Problems
5.1 Linear equations5.1.1 Gaussian elimination5.1.2 Gaussian elimination by hand5.1.3 When can we perform Gaussian elimination?5.1.4 The time complexity of Gaussian elimination5.1.5 When can a system of linear equations be solved?5.1.6 Inverting matrices5.2 The LU decomposition5.2.1 Implementing the LU decomposition5.2.2 Inverting a matrix, for real5.2.3 How to actually invert matrices5.3 Determinants in practice5.3.1 The lesser of two evils5.3.2 The recursive way5.3.3 How to actually compute determinants5.4 Summary5.5 Problems
6.1 Eigenvalues of matrices6.2 Finding eigenvalue-eigenvector pairs6.2.1 The characteristic polynomial6.2.2 Finding eigenvectors6.3 Eigenvectors, eigenspaces, and their bases6.4 Summary6.5 Problems
7.1 Special transformations7.1.1 The adjoint transformation7.1.2 Orthogonal transformations7.2 Self-adjoint transformations and the spectral decomposition theorem7.3 The singular value decomposition7.4 Orthogonal projections7.4.1 Properties of orthogonal projections7.4.2 Orthogonal projections are the optimal projections7.5 Computing eigenvalues7.5.1 Power iteration for calculating the eigenvectors of real symmetric matrices7.5.2 Power iteration in practice7.5.3 Power iteration for the rest of the eigenvectors7.6 The QR algorithm7.6.1 The QR decomposition7.6.2 Iterating the QR decomposition7.7 Summary7.8 Problems
8.1 The directed graph of a nonnegative matrix8.2 Benefits of the graph representation8.2.1 The connectivity of graphs8.3 The Frobenius normal form8.3.1 Permutation matrices8.3.2 Directed graphs and their strongly connected components8.3.3 Putting graphs and permutation matrices together8.4 Summary8.5 Problems

9.1 Functions in theory9.1.1 The mathematical definition of a function9.1.2 Domain and image9.1.3 Operations with functions9.1.4 Mental models of functions9.2 Functions in practice9.2.1 Operations on functions9.2.2 Functions as callable objects9.2.3 Function base class9.2.4 Composition in the object-oriented way9.3 Summary9.4 Problems
10.1 Numbers10.1.1 Natural numbers and integers10.1.2 Rational numbers10.1.3 Real numbers10.2 Sequences10.2.1 Convergence10.2.2 Properties of convergence10.2.3 Famous convergent sequences10.2.4 The role of convergence in machine learning10.2.5 Divergent sequences10.2.6 The big and small O notation10.2.7 Real numbers are sequences10.3 Series10.3.1 Convergent and divergent series10.3.2 Properties of series10.3.3 Conditional and absolute convergence10.3.4 Revisiting rearrangements10.3.5 Convergence tests for series10.3.6 The Cauchy product of series10.4 Summary10.5 Problems
11.1 Topology11.1.1 Open and closed sets11.1.2 Distance and topology11.1.3 Sets and sequences11.1.4 Bounded sets11.1.5 Compact sets11.2 Limits11.2.1 Equivalent definitions of limits11.3 Continuity11.3.1 Properties of continuous functions11.4 Summary11.5 Problems
12.1 Differentiation in theory12.1.1 Equivalent forms of differentiation12.1.2 Differentiation and continuity12.2 Differentiation in practice12.2.1 Rules of differentiation12.2.2 Derivatives of elementary functions12.2.3 Higher-order derivatives12.2.4 Extending the Function base class12.2.5 The derivative of compositions12.2.6 Numerical differentiation12.3 Summary12.4 Problems
13.1 Minima, maxima, and derivatives13.1.1 Local minima and maxima13.1.2 Characterization of optima with higher order derivatives13.1.3 Mean value theorems13.2 The basics of gradient descent13.2.1 Derivatives, revisited13.2.2 The gradient descent algorithm13.2.3 Implementing gradient descent13.2.4 Drawbacks and caveats13.3 Why does gradient descent work?13.3.1 Differential equations 10113.3.2 The (slightly more) general form of ODEs13.3.3 A geometric interpretation of differential equations13.3.4 A continuous version of gradient ascent13.3.5 Gradient ascent as a discretized differential equation13.3.6 Gradient ascent in action13.4 Summary13.5 Problems
14.1 Integration in theory14.1.1 Partitions and their refinements14.1.2 The Riemann integral14.1.3 Integration as the inverse of differentiation14.2 Integration in practice14.2.1 Integrals and operations14.2.2 Integration by parts14.2.3 Integration by substitution14.2.4 Numerical integration14.2.5 Implementing the trapezoidal rule14.3 Summary14.4 ProblemsJoin our community on Discord
15.1 What is a multivariable function?15.2 Linear functions in multiple variables15.3 The curse of dimensionality15.4 Summary
16.1 Partial and total derivatives16.1.1 The gradient16.1.2 Higher order partial derivatives16.1.3 The total derivative16.1.4 Directional derivatives16.1.5 Properties of the gradient16.2 Derivatives of vector-valued functions16.2.1 The derivatives of curves16.2.2 The Jacobian and Hessian matrices16.2.3 The total derivative for vector-vector functions16.2.4 Derivatives and function operations16.3 Summary16.4 Problems
17.1 Multivariable functions in code17.2 Minima and maxima, revisited17.3 Gradient descent in its full form17.4 Summary17.5 Problems
18.1 The language of thinking18.1.1 Thinking in absolutes18.1.2 Thinking in probabilities18.2 The axioms of probability18.2.1 Event spaces and σ-algebras18.2.2 Describing σ-algebras18.2.3 σ-algebras over real numbers18.2.4 Probability measures18.2.5 Fundamental properties of probability18.2.6 Probability spaces on ℝn18.2.7 How to interpret probability18.3 Conditional probability18.3.1 Independence18.3.2 The law of total probability revisited18.3.3 The Bayes theorem18.3.4 The Bayesian interpretation of probability18.3.5 The probabilistic inference process18.3.6 The Monty Hall paradox18.4 Summary18.5 Problems
19.1 Random variables19.1.1 Discrete random variables19.1.2 Real-valued random variables19.1.3 Random variables in general19.1.4 Behind the definition of random variables19.1.5 Independence of random variables19.2 Discrete distributions19.2.1 The Bernoulli distribution19.2.2 The binomial distribution19.2.3 The geometric distribution19.2.4 The uniform distribution19.2.5 The single-point distribution19.2.6 Law of total probability, revisited once more19.2.7 Sums of discrete random variables19.3 Real-valued distributions19.3.1 The cumulative distribution function19.3.2 Properties of the distribution function19.3.3 Cumulative distribution functions for discrete random variables19.3.4 The uniform distribution19.3.5 The exponential distribution19.3.6 The normal distribution19.4 Density functions19.4.1 Density functions in practice19.4.2 Classification of real-valued random variables19.5 Summary19.6 Problems
20.1 Discrete random variables20.1.1 The expected value in poker20.2 Continuous random variables20.3 Properties of the expected value20.4 Variance20.4.1 Covariance and correlation20.5 The law of large numbers20.5.1 Tossing coins…20.5.2 …rolling dice…20.5.3 …and all the rest20.5.4 The weak law of large numbers20.5.5 The strong law of large numbers20.6 Information theory20.6.1 Guess the number20.6.2 Guess the number 2: Electric Boogaloo20.6.3 Information and entropy20.6.4 Differential entropy20.7 The Maximum Likelihood Estimation20.7.1 Probabilistic modeling 10120.7.2 Modeling heights20.7.3 The general method20.7.4 The German tank problem20.8 Summary20.9 Problems
A.1 Mathematical logic 101A.2 Logical connectivesA.3 The propositional calculusA.4 Variables and predicatesA.5 Existential and universal quantificationA.6 Problems
B.1 What is a definition?B.2 What is a theorem?B.3 What is a proof?B.4 EquivalencesB.5 Proof techniquesB.5.1 Proof by inductionB.5.2 Proof by contradictionB.5.3 Contraposition
C.1 What is a set?C.2 Operations on setsC.2.1 Union, intersection, differenceC.2.2 De Morgan’s lawsC.3 The Cartesian productC.4 The cardinality of setsC.5 The Russell paradox (optional)
D.1 The definition of complex numbersD.2 The geometric representationD.3 The fundamental theorem of algebraD.4 Why are complex numbers important?

Content preview from Mathematics of Machine Learning

1 Vectors and Vector Spaces

”I want to point out that the class of abstract linear spaces is no larger than the class of spaces whose elements are arrays. So what is gained by abstraction? First of all, the freedom to use a single symbol for an array; this way we can think of vectors as basic building blocks, unencumbered by components. The abstract view leads to simple, transparent proofs of results.”

— Peter D. Lax, in Chapter 1 of his book Linear Algebra and its Applications

The mathematics of machine learning rests upon three pillars: linear algebra, calculus, and probability theory. Linear algebra describes how to represent and manipulate data; calculus helps us fit the models; while probability theory helps interpret them.

These build ...