Practical Linear Algebra for Data Science

Chapter 1. Introduction

What Is Linear Algebra and Why Learn It?

Linear algebra has an interesting history in mathematics, dating back to the 17th century in the West and much earlier in China. Matrices—the spreadsheets of numbers at the heart of linear algebra—were used to provide a compact notation for storing sets of numbers like geometric coordinates (this was Descartes’s original use of matrices) and systems of equations (pioneered by Gauss). In the 20th century, matrices and vectors were used for multivariate mathematics including calculus, differential equations, physics, and economics.

But most people didn’t need to care about matrices until fairly recently. Here’s the thing: computers are extremely efficient at working with matrices. And so, modern computing gave rise to modern linear algebra. Modern linear algebra is computational, whereas traditional linear algebra is abstract. Modern linear algebra is best learned through code and applications in graphics, statistics, data science, AI, and numerical simulations, whereas traditional linear algebra is learned through proofs and pondering infinite-dimensional vector spaces. Modern linear algebra provides the structural beams that support nearly every algorithm implemented on computers, whereas traditional linear algebra is often intellectual fodder for advanced mathematics university students.

Welcome to modern linear algebra.

Should you learn linear algebra? That depends on whether you want to understand algorithms and procedures, or simply apply methods that others have developed. I don’t mean to disparage the latter—there is nothing intrinsically wrong with using tools you don’t understand (I am writing this on a laptop that I can use but could not build from scratch). But given that you are reading a book with this title in the O’Reilly book collection, I guess you either (1) want to know how algorithms work or (2) want to develop or adapt computational methods. So yes, you should learn linear algebra, and you should learn the modern version of it.

About This Book

The purpose of this book is to teach you modern linear algebra. But this is not about memorizing some key equations and slugging through abstract proofs; the purpose is to teach you how to think about matrices, vectors, and operations acting upon them. You will develop a geometric intuition for why linear algebra is the way it is. And you will understand how to implement linear algebra concepts in Python code, with a focus on applications in machine learning and data science.

Many traditional linear algebra textbooks avoid numerical examples in the interest of generalizations, expect you to derive difficult proofs on your own, and teach myriad concepts that have little or no relevance to application or implementation in computers. I do not write these as criticisms—abstract linear algebra is beautiful and elegant. But if your goal is to use linear algebra (and mathematics more generally) as a tool for understanding data, statistics, deep learning, image processing, etc., then traditional linear algebra textbooks may seem like a frustrating waste of time that leave you confused and concerned about your potential in a technical field.

This book is written with self-studying learners in mind. Perhaps you have a degree in math, engineering, or physics, but need to learn how to implement linear algebra in code. Or perhaps you didn’t study math at university and now realize the importance of linear algebra for your studies or work. Either way, this book is a self-contained resource; it is not solely a supplement for a lecture-based course (though it could be used for that purpose).

If you were nodding your head in agreement while reading the past three paragraphs, then this book is definitely for you.

If you would like to take a deeper dive into linear algebra, with more proofs and explorations, then there are several excellent texts that you can consider, including my own Linear Algebra: Theory, Intuition, Code (Sincxpress BV).¹

Prerequisites

I have tried to write this book for enthusiastic learners with minimal formal background. That said, nothing is ever learned truly from from scratch.

Math

You need to be comfortable with high-school math. Just basic algebra and geometry; nothing fancy.

Absolutely zero calculus is required for this book (though differential calculus is important for applications where linear algebra is often used, such as deep learning and optimization).

But most importantly, you need to be comfortable thinking about math, looking at equations and graphics, and embracing the intellectual challenge that comes with studying math.

Attitude

Linear algebra is a branch of mathematics, ergo this is a mathematics book. Learning math, especially as an adult, requires some patience, dedication, and an assertive attitude. Get a cup of coffee, take a deep breath, put your phone in a different room, and dive in.

There will be a voice in the back of your head telling you that you are too old or too stupid to learn advanced mathematics. Sometimes that voice is louder and sometimes softer, but it’s always there. And it’s not just you—everyone has it. You cannot suppress or destroy that voice; don’t even bother trying. Just accept that a bit of insecurity and self-doubt is part of being human. Each time that voice speaks up is a challenge for you to prove it wrong.

Coding

This book is focused on linear algbera applications in code. I wrote this book for Python, because Python is currently the most widely used language in data science, machine learning, and related fields. If you prefer other languages like MATLAB, R, C, or Julia, then I hope you find it straightforward to translate the Python code.

I’ve tried to make the Python code as simple as possible, while still being relevant for applications. Chapter 16 provides a basic introduction to Python programming. Should you go through that chapter? That depends on your level of Python skills:

Intermediate/advanced (>1 year coding experience)

Skip Chapter 16 entirely, or perhaps skim it to get a sense of the kind of code that will appear in the rest of the book.

Some knowledge (<1 year experience)

Please work through the chapter in case there is material that is new or that you need to refresh. But you should be able to get through it rather briskly.

Total beginner

Go through the chapter in detail. Please understand that this book is not a complete Python tutorial, so if you find yourself struggling with the code in the content chapters, you might want to put this book down, work through a dedicated Python course or book, then come back to this book.

Mathematical Proofs Versus Intuition from Coding

The purpose of studying math is, well, to understand math. How do you understand math? Let us count the ways:

Rigorous proofs

A proof in mathematics is a sequence of statements showing that a set of assumptions leads to a logical conclusion. Proofs are unquestionably important in pure mathematics.

Visualizations and examples

Clearly written explanations, diagrams, and numerical examples help you gain intuition for concepts and operations in linear algebra. Most examples are done in 2D or 3D for easy visualization, but the principles also apply to higher dimensions.

The difference between these is that formal mathematical proofs provide rigor but rarely intuition, whereas visualizations and examples provide lasting intuition through hands-on experience but can risk inaccuracies based on specific examples that do not generalize.

Proofs of important claims are included, but I focus more on building intuition through explanations, visualizations, and code examples.

And this brings me to mathematical intuition from coding (what I sometimes call “soft proofs”). Here’s the idea: you assume that Python (and libraries such as NumPy and SciPy) correctly implements the low-level number crunching, while you focus on the principles by exploring many numerical examples in code.

A quick example: we will “soft-prove” the commutivity principle of multiplication, which states that $a\times b=b\times a$:

a = np.random.randn()
b = np.random.randn()
a*b - b*a

This code generates two random numbers and tests the hypothesis that swapping the order of multiplication has no impact on the result. The third line of would print out 0.0 if the commutivity principle is true. If you run this code multiple times and always get 0.0, then you have gained intuition for commutivity by seeing the same result in many different numerical examples.

To be clear: intuition from code is no substitute for a rigorous mathematical proof. The point is that “soft proofs” allow you to understand mathematical concepts without having to worry about the details of abstract mathematical syntax and arguments. This is particularly advantageous to coders who lack an advanced mathematics background.

The bottom line is that you can learn a lot of math with a bit of coding.

Code, Printed in the Book and Downloadable Online

You can read this book without looking at code or solving code exercises. That’s fine, and you will certainly learn something. But don’t be disappointed if your knowledge is superficial and fleeting. If you really want to understand linear algebra, you need to solve problems. That’s why this book comes with code demonstrations and exercises for each mathematical concept.

Important code is printed directly in the book. I want you to read the text and equations, look at the graphs, and see the code at the same time. That will allow you to link concepts and equations to code.

But printing code in a book can take up a lot of space, and hand-copying code on your computer is tedious. Therefore, only the key code lines are printed in the book pages; the online code contains additional code, comments, graphics embellishments, and so on. The online code also contains solutions to the coding exercises (all of them, not only the odd-numbered problems!). You should definitely download the code and go through it while working through the book.

All the code can be obtained from the GitHub site https://github.com/mikexco⁠hen​/LinAlg4DataScience. You can clone this repository or simply download the entire repository as a ZIP file (you do not need to register, log in, or pay to download the code).

I wrote the code using Jupyter notebook in Google’s Colab environment. I chose to use Jupyter because it’s a friendly and easy-to-use environment. That said, I encourage you to use whichever Python IDE you prefer. The online code is also provided as raw .py files for convenience.

Code Exercises

Math is not a spectator sport. Most math books have countless paper-and-pencil problems to work through (and let’s be honest: no one does all of them). But this book is all about applied linear algebra, and no one applies linear algebra on paper! Instead, you apply linear algebra in code. Therefore, in lieu of hand-worked problems and tedious proofs “left as an exercise to the reader” (as math textbook authors love to write), this book has lots of code exercises.

The code exercises vary in difficulty. If you are new to Python and to linear algebra, you might find some exercises really challenging. If you get stuck, here’s a suggestion: have a quick glance at my solution for inspiration, then put it away so you can’t see my code, and continue working on your own code.

When comparing your solution to mine, keep in mind that there are many ways to solve problems in Python. Ariving at the correct answer is important; the steps you take to get there are often a matter of personal coding style.

How to Use This Book (for Teachers and Self Learners)

There are three environments in which this book is useful:

Self-learner

I have tried to make this book accessible to readers who want to learn linear algebra on their own, outside a formal classroom environment. No additional resources or online lectures are necessary, although of course there are myriad other books, websites, YouTube videos, and online courses that students might find helpful.

Primary textbook in a data science class

This book can be used as a primary textbook in a course on the math underlying data science, machine learning, AI, and related topics. There are 14 content chapters (excluding this introduction and the Python appendix), and students could be expected to work though one to two chapters per week. Because students have access to the solutions to all exercises, instructors may wish to supplement the book exercises with additional problem sets.

Secondary textbook in a math-focused linear algebra course

This book could also be used as a supplement in a mathematics course with a strong focus on proofs. In this case, the lectures would focus on theory and rigorous proofs while this book could be referenced for translating the concepts into code with an eye towards applications in data science and machine learning. As I wrote above, instructors may wish to provide supplementary exercises because the solutions to all the book exercises are available online.

¹ Apologies for the shameless self-promotion; I promise that’s the only time in this book I’ll subject you to such an indulgence.