book

The Art of Machine Learning

Name: The Art of Machine Learning
Author: Norman Matloff
ISBN: 9781718502109

by Norman Matloff

January 2024

Beginner to intermediate

272 pages

6h 25m

English

No Starch Press

Read now

Unlock full access

Cover Page
Title Page
Copyright Page
About the Author
About the Technical Reviewer
BRIEF CONTENTS
CONTENTS IN DETAIL
ACKNOWLEDGMENTS
INTRODUCTION
0.1 What Is ML?0.2 The Role of Math in ML Theory and Practice0.3 Why Another ML Book?0.4 Recurring Special Sections0.5 Background Needed0.6 The qe*-Series Software0.7 The Book’s Grand Plan0.8 One More Point
PART I: PROLOGUE, AND NEIGHBORHOOD-BASED METHODS

1 REGRESSION MODELS
1.1 Example: The Bike Sharing Dataset1.1.1 Loading the Data1.1.2 A Look Ahead1.2 Machine Learning and Prediction1.2.1 Predicting Past, Present, and Future1.2.2 Statistics vs. Machine Learning in Prediction1.3 Introducing the k-Nearest Neighbors Method1.3.1 Predicting Bike Ridership with k-NN1.4 Dummy Variables and Categorical Variables1.5 Analysis with qeKNN()1.5.1 Predicting Bike Ridership with qeKNN()1.6 The Regression Function: The Basis of ML1.7 The Bias-Variance Trade-off1.7.1 Analogy to Election Polls1.7.2 Back to ML1.8 Example: The mlb Dataset1.9 k-NN and Categorical Features1.10 Scaling1.11 Choosing Hyperparameters1.11.1 Predicting the Training Data1.12 Holdout Sets1.12.1 Loss Functions1.12.2 Holdout Sets in the qe*-Series1.12.3 Motivating Cross-Validation1.12.4 Hyperparameters, Dataset Size, and Number of Features1.13 Pitfall: p-Hacking and Hyperparameter Selection1.14 Pitfall: Long-Term Time Trends1.15 Pitfall: Dirty Data1.16 Pitfall: Missing Data1.17 Direct Access to the regtools k-NN Code1.18 Conclusions
2 CLASSIFICATION MODELS
2.1 Classification Is a Special Case of Regression2.2 Example: The Telco Churn Dataset2.2.1 Pitfall: Factor Data Read as Non-factor2.2.2 Pitfall: Retaining Useless Features2.2.3 Dealing with NA Values2.2.4 Applying the k-Nearest Neighbors Method2.2.5 Pitfall: Overfitting Due to Features with Many Categories2.3 Example: Vertebrae Data2.3.1 Analysis2.4 Pitfall: Error Rate Improves Only Slightly Using the Features2.5 The Confusion Matrix2.6 Clearing the Confusion: Unbalanced Data2.6.1 Example: The Kaggle Appointments Dataset2.6.2 A Better Approach to Unbalanced Data2.7 Receiver Operating Characteristic and Area Under Curve2.7.1 Details of ROC and AUC2.7.2 The qeROC() Function2.7.3 Example: Telco Churn Data2.7.4 Example: Vertebrae Data2.7.5 Pitfall: Overreliance on AUC2.8 Conclusions
3 BIAS, VARIANCE, OVERFITTING, AND CROSS-VALIDATION
3.1 Overfitting and Underfitting3.1.1 Intuition Regarding the Number of Features and Overfitting3.1.2 Relation to Overall Dataset Size3.1.3 Well Then, What Are the Best Values of k and p?3.2 Cross-Validation3.2.1 K-Fold Cross-Validation3.2.2 Using the replicMeans() Function3.2.3 Example: Programmer and Engineer Data3.2.4 Triple Cross-Validation3.3 Conclusions
4 DEALING WITH LARGE NUMBERS OF FEATURES
4.1 Pitfall: Computational Issues in Large Datasets4.2 Introduction to Dimension Reduction4.2.1 Example: The Million Song Dataset4.2.2 The Need for Dimension Reduction4.3 Methods for Dimension Reduction4.3.1 Consolidation and Embedding4.3.2 The All Possible Subsets Method4.3.3 Principal Components Analysis4.3.4 But Now We Have Two Hyperparameters4.3.5 Using the qePCA() Wrapper4.3.6 PCs and the Bias-Variance Trade-off4.4 The Curse of Dimensionality4.5 Other Methods of Dimension Reduction4.5.1 Feature Ordering by Conditional Independence4.5.2 Uniform Manifold Approximation and Projection4.6 Going Further Computationally4.7 Conclusions
PART II: TREE-BASED METHODS
5 A STEP BEYOND K-NN: DECISION TREES
5.1 Basics of Decision Trees5.2 The qeDT() Function5.2.1 Looking at the Plot5.3 Example: New York City Taxi Data5.3.1 Pitfall: Too Many Combinations of Factor Levels5.3.2 Tree-Based Analysis5.4 Example: Forest Cover Data5.5 Decision Tree Hyperparameters: How to Split?5.6 Hyperparameters in the qeDT() Function5.7 Conclusions
6 TWEAKING THE TREES
6.1 Bias vs. Variance, Bagging, and Boosting6.2 Bagging: Generating New Trees by Resampling6.2.1 Random Forests6.2.2 The qeRF() Function6.2.3 Example: Vertebrae Data6.2.4 Example: Remote-Sensing Soil Analysis6.3 Boosting: Repeatedly Tweaking a Tree6.3.1 Implementation: AdaBoost6.3.2 Gradient Boosting6.3.3 Example: Call Network Monitoring6.3.4 Example: Vertebrae Data6.3.5 Bias vs. Variance in Boosting6.3.6 Computational Speed6.3.7 Further Hyperparameters6.3.8 The Learning Rate6.4 Pitfall: No Free Lunch
7 FINDING A GOOD SET OF HYPERPARAMETERS
7.1 Combinations of Hyperparameters7.2 Grid Searching with qeFT()7.2.1 How to Call qeFT()7.3 Example: Programmer and Engineer Data7.3.1 Confidence Intervals7.3.2 The Takeaway on Grid Searching7.4 Example: Programmer and Engineer Data7.5 Example: Phoneme Data7.6 Conclusions
PART III: METHODS BASED ON LINEAR RELATIONSHIPS
8 PARAMETRIC METHODS
8.1 Motivating Example: The Baseball Player Data8.1.1 A Graph to Guide Our Intuition8.1.2 View as Dimension Reduction8.2 The lm() Function8.3 Wrapper for lm() in the qe*-Series: qeLin()8.4 Use of Multiple Features8.4.1 Example: Baseball Player, Continued8.4.2 Beta Notation8.4.3 Example: Airbnb Data8.4.4 Applying the Linear Model8.5 Dimension Reduction8.5.1 Which Features Are Important?8.5.2 Statistical Significance and Dimension Reduction8.6 Least Squares and Residuals8.7 Diagnostics: Is the Linear Model Valid?8.7.1 Exactness?8.7.2 Diagnostic Methods8.8 The R-Squared Value(s)8.9 Classification Applications: The Logistic Model8.9.1 The glm() and qeLogit() Functions8.9.2 Example: Telco Churn Data8.9.3 Multiclass Case8.9.4 Example: Fall Detection Data8.10 Bias and Variance in Linear/Generalized Linear Models8.10.1 Example: Bike Sharing Data8.11 Polynomial Models8.11.1 Motivation8.11.2 Modeling Nonlinearity with a Linear Model8.11.3 Polynomial Logistic Regression8.11.4 Example: Programmer and Engineer Wages8.12 Blending the Linear Model with Other Methods8.13 The qeCompare() Function8.13.1 Need for Caution Regarding Polynomial Models8.14 What’s Next
9 CUTTING THINGS DOWN TO SIZE: REGULARIZATION
9.1 Motivation9.2 Size of a Vector9.3 Ridge Regression and the LASSO9.3.1 How They Work9.3.2 The Bias-Variance Trade-off, Avoiding Overfitting9.3.3 Relation Between λ, n, and p9.3.4 Comparison, Ridge vs. LASSO9.4 Software9.5 Example: NYC Taxi Data9.6 Example: Airbnb Data9.7 Example: African Soil Data9.7.1 LASSO Analysis9.8 Optional Section: The Famous LASSO Picture9.9 Coming Up
PART IV: METHODS BASED ON SEPARATING LINES AND PLANES
10 A BOUNDARY APPROACH: SUPPORT VECTOR MACHINES
10.1 Motivation10.1.1 Example: The Forest Cover Dataset10.2 Lines, Planes, and Hyperplanes10.3 Math Notation10.3.1 Vector Expressions10.3.2 Dot Products10.3.3 SVM as a Parametric Model10.4 SVM: The Basic Ideas—Separable Case10.4.1 Example: The Anderson Iris Dataset10.4.2 Optimizing Criterion10.5 Major Problem: Lack of Linear Separability10.5.1 Applying a “Kernel”10.5.2 Soft Margin10.6 Example: Forest Cover Data10.7 And What About That Kernel Trick?10.8 “Warning: Maximum Number of Iterations Reached”10.9 Summary
11 LINEAR MODELS ON STEROIDS: NEURAL NETWORKS
11.1 Overview11.2 Working on Top of a Complex Infrastructure11.3 Example: Vertebrae Data11.4 Neural Network Hyperparameters11.5 Activation Functions11.6 Regularization11.6.1 L1 and L2 Regularization11.6.2 Regularization by Dropout11.7 Example: Fall Detection Data11.8 Pitfall: Convergence Problems11.9 Close Relation to Polynomial Regression11.10 Bias vs. Variance in Neural Networks11.11 Discussion
PART V: APPLICATIONS
12 IMAGE CLASSIFICATION
12.1 Example: The Fashion MNIST Data12.1.1 A First Try Using a Logit Model12.1.2 Refinement via PCA12.2 Convolutional Models12.2.1 Need for Recognition of Locality12.2.2 Overview of Convolutional Methods12.2.3 Image Tiling12.2.4 The Convolution Operation12.2.5 The Pooling Operation12.2.6 Shape Evolution Across Layers12.2.7 Dropout12.2.8 Summary of Shape Evolution12.2.9 Translation Invariance12.3 Tricks of the Trade12.3.1 Data Augmentation12.3.2 Pretrained Networks12.4 So, What About the Overfitting Issue?12.5 Conclusions
13 HANDLING TIME SERIES AND TEXT DATA
13.1 Converting Time Series Data to Rectangular Form13.1.1 Toy Example13.1.2 The regtools Function TStoX()13.2 The qeTS() Function13.3 Example: Weather Data13.4 Bias vs. Variance13.5 Text Applications13.5.1 The Bag-of-Words Model13.5.2 The qeText() Function13.5.3 Example: Quiz Data13.5.4 Example: AG News Dataset13.6 Summary
A LIST OF ACRONYMS AND SYMBOLS
B STATISTICS AND ML TERMINOLOGY CORRESPONDENCE
C MATRICES, DATA FRAMES, AND FACTOR CONVERSIONS
C.1 MatricesC.2 Conversions: Between R Factors and Dummy Variables, Between Data Frames and Matrices
D PITFALL: BEWARE OF “P-HACKING”!
INDEX

Content preview from The Art of Machine Learning

3BIAS, VARIANCE, OVERFITTING, AND CROSS-VALIDATION

We now look in detail at a vital topic touched on in Sections 1.7, 1.12.4, and 2.2.5—overfitting. In this chapter, we’ll explain what bias and variance really mean in ML contexts and how they affect overfitting. We’ll then cover a popular approach to avoiding overfitting known as cross-validation.

The problem of overfitting exemplifies the point made in the title of this book: ML is an art, not a science. There is no formulaic solution to various problems, especially overfitting. Professor Yaser Abu-Mostafa of Caltech, a prominent ML figure, once summed it up: “The ability to avoid overfitting ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098168759Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

The Art of Machine Learning

by Norman Matloff

3BIAS, VARIANCE, OVERFITTING, AND CROSS-VALIDATION

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.