book

Machine Learning Foundations, Volume 1: Supervised Learning

by Roi Yehoshua

September 2025

Intermediate to advanced

812 pages

23h 14m

English

Addison-Wesley Professional

Read now

Unlock full access

1.1 Formal Definition1.2 Types of Machine Learning1.3 Related Fields1.4 Brief History1.5 Machine Learning Applications1.6 Limitations of Machine Learning1.7 Ethical Considerations1.8 Software Libraries1.9 Common Datasets1.10 Current Trends and Future Directions1.11 Summary
2.1 Formal Definition2.2 Machine Learning Models2.3 The Data-Generating Process2.4 Maximum Likelihood Estimation2.5 Bayesian Statistics2.6 The Bias-Variance Tradeoff2.7 Building a Machine Learning Model2.8 Challenges in Supervised Learning2.9 Summary2.10 Exercises
3.1 Main Features3.2 Installation3.3 The Estimator API3.4 Typical Workflow of Building a Model3.5 Example: Iris Classification3.6 Pipelines3.7 Cross-Validation3.8 Hyperparameter Tuning3.9 Data Preparation3.10 Data Preprocessing3.11 Building Your Own Estimators3.12 Summary3.13 Exercises
4.1 Formal Definitions and Notations4.2 Ordinary Least Squares (OLS)4.3 Simple Linear Regression4.4 Multiple Linear Regression4.5 Regression Evaluation Metrics4.6 Example: Predicting House Prices4.7 Linear Regression Assumptions4.8 Gradient Descent4.9 Alternative Loss Functions4.10 Non-Linear Regression4.11 Regularized Linear Regression4.12 (*) Bayesian Linear Regression4.13 Summary4.14 Exercises

5.1 Classification Problems5.2 The Logistic Regression Model5.3 Training a Logistic Regression Model5.4 Logistic Regression in Scikit-Learn5.5 Classification Evaluation Metrics5.6 Learning from Imbalanced Data5.7 Multi-Class Extensions for Binary Classifiers5.8 Multinomial Logistic Regression5.9 (*) Generalized Linear Models5.10 Summary5.11 Exercises
6.1 K-Nearest Neighbors Classification6.2 Choosing the Number of Neighbors6.3 Distance Metrics and Similarity Measures6.4 (*) KNN Classification Error Bounds6.5 Efficient Data Structures for KNN6.6 The Curse of Dimensionality6.7 Radius-based Nearest Neighbors6.8 K-Nearest Neighbors Regression6.9 Variants of KNN6.10 Applications of KNN6.11. Summary6.12 Exercises
7.1 The Naive Bayes Model7.2 Event Models in Naive Bayes7.3 Naive Bayes Classifiers in Scikit-Learn7.4 (*) Linear and Quadratic Discriminant Analysis7.5 Introduction to Natural Language Processing7.6 Document Classification Example7.7 Bayesian Networks7.8 Summary7.9 Exercises
8.1 Decision Tree Definition8.2 Decision Tree Construction8.3 Tree Pruning8.4 Decision Tree Algorithms8.5 Decision Trees in Scikit-Learn8.6 Regression Trees8.7 Multi-Output Problems8.8 Oblique Decision Trees8.9 Summary8.10 Exercises
9.1 Introduction and Motivation9.2 Types of Ensemble Methods9.3 Using Different Learning Algorithms9.4 Bagging Ensembles9.5 Random Forests9.6 Boosting Ensembles9.7 AdaBoost9.8 Gradient Boosting9.9 Stacking Ensembles9.10 Summary9.11 Exercises
10.1 XGBoost10.2 LightGBM10.3 CatBoost10.4 The Higgs Boson Machine Learning Challenge10.5 Summary10.6 Exercises
11.1 Hard-Margin SVM11.2 Soft-Margin SVM11.3 Kernel Methods11.4 Non-Linear SVM11.5 ν-SVM11.6 Support Vector Regression (SVR)11.7 (*) Efficient Methods for Training SVMs11.8 Model Calibration11.9 Summary11.10 Exercises
12.1 Supervised Learning Summary12.2 Choosing a Learning Algorithm12.3 Conducting Machine Learning Experiments12.4 Research in Machine Learning12.5 Machine Learning Competitions12.6 Additional Resources12.7 Exercises

Content preview from Machine Learning Foundations, Volume 1: Supervised Learning

Chapter 8. Decision Trees

Decision trees are powerful and versatile machine learning models that use a sequence of simple decision rules, organized in a tree-like structure, to make predictions or classify data points. As non-parametric models, they do not rely on any assumption about the data’s distribution, making them suitable for a wide variety of data types and distributions.

One of the key advantages of decision trees is their interpretability. The ability to visualize the tree makes the decision-making process of the model transparent and easy to understand, even for non-experts. This transparency is essential in fields such as medicine, where decision trees help diagnose diseases based on patient symptoms and test results, and finance, ...