book

Mathematics of Machine Learning

Name: Mathematics of Machine Learning
Author: Tivadar Danka
ISBN: 9781837027873

by Tivadar Danka

May 2025

Intermediate to advanced

730 pages

20h 14m

English

Packt Publishing

Read now

Unlock full access

Introduction
What is this book about?How to read this bookConventions usedWhat this book coversTo get the most out of this book
Part 1: Linear Algebra
1 Vectors and Vector Spaces
1.1 What is a vector space?1.1.1 Examples of vector spaces1.2 The basis1.2.1 Linear combinations and independence1.2.2 Spans of vector sets1.2.3 Bases, the minimal generating sets1.2.4 Finite dimensional vector spaces1.2.5 Why are bases so important?1.2.6 The existence of bases1.2.7 Subspaces1.3 Vectors in practice1.3.1 Tuples1.3.2 Lists1.3.3 NumPy arrays1.3.4 NumPy arrays as vectors1.3.5 Is NumPy really faster than Python?1.4 Summary1.5 Problems
2 The Geometric Structure of Vector Spaces
2.1 Norms and distances2.1.1 Defining distances from norms2.2 Inner products, angles, and lots of reasons to care about them2.2.1 The generated norm2.2.2 Orthogonality2.2.3 The geometric interpretation of inner products2.2.4 Orthogonal and orthonormal bases2.2.5 The Gram-Schmidt orthogonalization process2.2.6 The orthogonal complement2.3 Summary2.4 Problems
3 Linear Algebra in Practice
3.1 Vectors in NumPy3.1.1 Norms, distances, and dot products3.1.2 The Gram-Schmidt orthogonalization process3.2 Matrices, the workhorses of linear algebra3.2.1 Manipulating matrices3.2.2 Matrices as arrays3.2.3 Matrices in NumPy3.2.4 Matrix multiplication, revisited3.2.5 Matrices and data3.3 Summary3.4 Problems
4 Linear Transformations
4.1 What is a linear transformation?4.1.1 Linear transformations and matrices4.1.2 Matrix operations revisited4.1.3 Inverting linear transformations4.1.4 The kernel and the image4.2 Change of basis4.2.1 The transformation matrix4.3 Linear transformations in the Euclidean plane4.3.1 Stretching4.3.2 Rotations4.3.3 Shearing4.3.4 Reflection4.3.5 Orthogonal projection4.4 Determinants, or how linear transformations affect volume4.4.1 How linear transformations scale the area4.4.2 The multi-linearity of determinants4.4.3 Fundamental properties of the determinants4.5 Summary4.6 Problems
5 Matrices and Equations
5.1 Linear equations5.1.1 Gaussian elimination5.1.2 Gaussian elimination by hand5.1.3 When can we perform Gaussian elimination?5.1.4 The time complexity of Gaussian elimination5.1.5 When can a system of linear equations be solved?5.1.6 Inverting matrices5.2 The LU decomposition5.2.1 Implementing the LU decomposition5.2.2 Inverting a matrix, for real5.2.3 How to actually invert matrices5.3 Determinants in practice5.3.1 The lesser of two evils5.3.2 The recursive way5.3.3 How to actually compute determinants5.4 Summary5.5 Problems
6 Eigenvalues and Eigenvectors
6.1 Eigenvalues of matrices6.2 Finding eigenvalue-eigenvector pairs6.2.1 The characteristic polynomial6.2.2 Finding eigenvectors6.3 Eigenvectors, eigenspaces, and their bases6.4 Summary6.5 Problems
7 Matrix Factorizations
7.1 Special transformations7.1.1 The adjoint transformation7.1.2 Orthogonal transformations7.2 Self-adjoint transformations and the spectral decomposition theorem7.3 The singular value decomposition7.4 Orthogonal projections7.4.1 Properties of orthogonal projections7.4.2 Orthogonal projections are the optimal projections7.5 Computing eigenvalues7.5.1 Power iteration for calculating the eigenvectors of real symmetric matrices7.5.2 Power iteration in practice7.5.3 Power iteration for the rest of the eigenvectors7.6 The QR algorithm7.6.1 The QR decomposition7.6.2 Iterating the QR decomposition7.7 Summary7.8 Problems
8 Matrices and Graphs
8.1 The directed graph of a nonnegative matrix8.2 Benefits of the graph representation8.2.1 The connectivity of graphs8.3 The Frobenius normal form8.3.1 Permutation matrices8.3.2 Directed graphs and their strongly connected components8.3.3 Putting graphs and permutation matrices together8.4 Summary8.5 Problems

References
Part 2: Calculus
9 Functions
9.1 Functions in theory9.1.1 The mathematical definition of a function9.1.2 Domain and image9.1.3 Operations with functions9.1.4 Mental models of functions9.2 Functions in practice9.2.1 Operations on functions9.2.2 Functions as callable objects9.2.3 Function base class9.2.4 Composition in the object-oriented way9.3 Summary9.4 Problems
10 Numbers, Sequences, and Series
10.1 Numbers10.1.1 Natural numbers and integers10.1.2 Rational numbers10.1.3 Real numbers10.2 Sequences10.2.1 Convergence10.2.2 Properties of convergence10.2.3 Famous convergent sequences10.2.4 The role of convergence in machine learning10.2.5 Divergent sequences10.2.6 The big and small O notation10.2.7 Real numbers are sequences10.3 Series10.3.1 Convergent and divergent series10.3.2 Properties of series10.3.3 Conditional and absolute convergence10.3.4 Revisiting rearrangements10.3.5 Convergence tests for series10.3.6 The Cauchy product of series10.4 Summary10.5 Problems
11 Topology, Limits, and Continuity
11.1 Topology11.1.1 Open and closed sets11.1.2 Distance and topology11.1.3 Sets and sequences11.1.4 Bounded sets11.1.5 Compact sets11.2 Limits11.2.1 Equivalent definitions of limits11.3 Continuity11.3.1 Properties of continuous functions11.4 Summary11.5 Problems
12 Differentiation
12.1 Differentiation in theory12.1.1 Equivalent forms of differentiation12.1.2 Differentiation and continuity12.2 Differentiation in practice12.2.1 Rules of differentiation12.2.2 Derivatives of elementary functions12.2.3 Higher-order derivatives12.2.4 Extending the Function base class12.2.5 The derivative of compositions12.2.6 Numerical differentiation12.3 Summary12.4 Problems
13 Optimization
13.1 Minima, maxima, and derivatives13.1.1 Local minima and maxima13.1.2 Characterization of optima with higher order derivatives13.1.3 Mean value theorems13.2 The basics of gradient descent13.2.1 Derivatives, revisited13.2.2 The gradient descent algorithm13.2.3 Implementing gradient descent13.2.4 Drawbacks and caveats13.3 Why does gradient descent work?13.3.1 Differential equations 10113.3.2 The (slightly more) general form of ODEs13.3.3 A geometric interpretation of differential equations13.3.4 A continuous version of gradient ascent13.3.5 Gradient ascent as a discretized differential equation13.3.6 Gradient ascent in action13.4 Summary13.5 Problems
14 Integration
14.1 Integration in theory14.1.1 Partitions and their refinements14.1.2 The Riemann integral14.1.3 Integration as the inverse of differentiation14.2 Integration in practice14.2.1 Integrals and operations14.2.2 Integration by parts14.2.3 Integration by substitution14.2.4 Numerical integration14.2.5 Implementing the trapezoidal rule14.3 Summary14.4 ProblemsJoin our community on Discord
References
Part 3: Multivariable Calculus
15 Multivariable Functions
15.1 What is a multivariable function?15.2 Linear functions in multiple variables15.3 The curse of dimensionality15.4 Summary
16 Derivatives and Gradients
16.1 Partial and total derivatives16.1.1 The gradient16.1.2 Higher order partial derivatives16.1.3 The total derivative16.1.4 Directional derivatives16.1.5 Properties of the gradient16.2 Derivatives of vector-valued functions16.2.1 The derivatives of curves16.2.2 The Jacobian and Hessian matrices16.2.3 The total derivative for vector-vector functions16.2.4 Derivatives and function operations16.3 Summary16.4 Problems
17 Optimization in Multiple Variables
17.1 Multivariable functions in code17.2 Minima and maxima, revisited17.3 Gradient descent in its full form17.4 Summary17.5 Problems
References
Part 4: Probability Theory
18 What is Probability?
18.1 The language of thinking18.1.1 Thinking in absolutes18.1.2 Thinking in probabilities18.2 The axioms of probability18.2.1 Event spaces and σ-algebras18.2.2 Describing σ-algebras18.2.3 σ-algebras over real numbers18.2.4 Probability measures18.2.5 Fundamental properties of probability18.2.6 Probability spaces on ℝn18.2.7 How to interpret probability18.3 Conditional probability18.3.1 Independence18.3.2 The law of total probability revisited18.3.3 The Bayes theorem18.3.4 The Bayesian interpretation of probability18.3.5 The probabilistic inference process18.3.6 The Monty Hall paradox18.4 Summary18.5 Problems
19 Random Variables and Distributions
19.1 Random variables19.1.1 Discrete random variables19.1.2 Real-valued random variables19.1.3 Random variables in general19.1.4 Behind the definition of random variables19.1.5 Independence of random variables19.2 Discrete distributions19.2.1 The Bernoulli distribution19.2.2 The binomial distribution19.2.3 The geometric distribution19.2.4 The uniform distribution19.2.5 The single-point distribution19.2.6 Law of total probability, revisited once more19.2.7 Sums of discrete random variables19.3 Real-valued distributions19.3.1 The cumulative distribution function19.3.2 Properties of the distribution function19.3.3 Cumulative distribution functions for discrete random variables19.3.4 The uniform distribution19.3.5 The exponential distribution19.3.6 The normal distribution19.4 Density functions19.4.1 Density functions in practice19.4.2 Classification of real-valued random variables19.5 Summary19.6 Problems
20 The Expected Value
20.1 Discrete random variables20.1.1 The expected value in poker20.2 Continuous random variables20.3 Properties of the expected value20.4 Variance20.4.1 Covariance and correlation20.5 The law of large numbers20.5.1 Tossing coins…20.5.2 …rolling dice…20.5.3 …and all the rest20.5.4 The weak law of large numbers20.5.5 The strong law of large numbers20.6 Information theory20.6.1 Guess the number20.6.2 Guess the number 2: Electric Boogaloo20.6.3 Information and entropy20.6.4 Differential entropy20.7 The Maximum Likelihood Estimation20.7.1 Probabilistic modeling 10120.7.2 Modeling heights20.7.3 The general method20.7.4 The German tank problem20.8 Summary20.9 Problems
References
Part 5: Appendix
Appendix A It’s Just Logic
A.1 Mathematical logic 101A.2 Logical connectivesA.3 The propositional calculusA.4 Variables and predicatesA.5 Existential and universal quantificationA.6 Problems
Appendix B The Structure of Mathematics
B.1 What is a definition?B.2 What is a theorem?B.3 What is a proof?B.4 EquivalencesB.5 Proof techniquesB.5.1 Proof by inductionB.5.2 Proof by contradictionB.5.3 Contraposition
Appendix C Basics of Set Theory
C.1 What is a set?C.2 Operations on setsC.2.1 Union, intersection, differenceC.2.2 De Morgan’s lawsC.3 The Cartesian productC.4 The cardinality of setsC.5 The Russell paradox (optional)
Appendix D Complex Numbers
D.1 The definition of complex numbersD.2 The geometric representationD.3 The fundamental theorem of algebraD.4 Why are complex numbers important?
Other Books You May Enjoy
Index

Content preview from Mathematics of Machine Learning

17 Optimization in Multiple Variables

Hey! We are at the last checkpoint of our calculus study. What’s missing? Gradient descent, of course.

In the previous eight chapters, we lined up all of our ducks in a row, and now it’s time to take that shot. First, we’ll put multivariable functions to code. Previously, we built a convenient interface in the form of our Function class to represent differentiable functions. After the lengthy setup in the previous chapter, we can easily extend it, and with the power of vectorization, we don’t even have to change that much. Let’s go!

17.1 Multivariable functions in code

It’s been a long time since we put theory into code. So, let’s take a look at multivariable functions!

Last time, we built a Function ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781837027873

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mathematics of Machine Learning

by Tivadar Danka

17 Optimization in Multiple Variables

17.1 Multivariable functions in code

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.