book

Machine Learning Foundations, Volume 1: Supervised Learning

by Roi Yehoshua

September 2025

Intermediate to advanced

812 pages

23h 14m

English

Addison-Wesley Professional

Read now

Unlock full access

1.1 Formal Definition1.2 Types of Machine Learning1.3 Related Fields1.4 Brief History1.5 Machine Learning Applications1.6 Limitations of Machine Learning1.7 Ethical Considerations1.8 Software Libraries1.9 Common Datasets1.10 Current Trends and Future Directions1.11 Summary
2.1 Formal Definition2.2 Machine Learning Models2.3 The Data-Generating Process2.4 Maximum Likelihood Estimation2.5 Bayesian Statistics2.6 The Bias-Variance Tradeoff2.7 Building a Machine Learning Model2.8 Challenges in Supervised Learning2.9 Summary2.10 Exercises
3.1 Main Features3.2 Installation3.3 The Estimator API3.4 Typical Workflow of Building a Model3.5 Example: Iris Classification3.6 Pipelines3.7 Cross-Validation3.8 Hyperparameter Tuning3.9 Data Preparation3.10 Data Preprocessing3.11 Building Your Own Estimators3.12 Summary3.13 Exercises
4.1 Formal Definitions and Notations4.2 Ordinary Least Squares (OLS)4.3 Simple Linear Regression4.4 Multiple Linear Regression4.5 Regression Evaluation Metrics4.6 Example: Predicting House Prices4.7 Linear Regression Assumptions4.8 Gradient Descent4.9 Alternative Loss Functions4.10 Non-Linear Regression4.11 Regularized Linear Regression4.12 (*) Bayesian Linear Regression4.13 Summary4.14 Exercises

5.1 Classification Problems5.2 The Logistic Regression Model5.3 Training a Logistic Regression Model5.4 Logistic Regression in Scikit-Learn5.5 Classification Evaluation Metrics5.6 Learning from Imbalanced Data5.7 Multi-Class Extensions for Binary Classifiers5.8 Multinomial Logistic Regression5.9 (*) Generalized Linear Models5.10 Summary5.11 Exercises
6.1 K-Nearest Neighbors Classification6.2 Choosing the Number of Neighbors6.3 Distance Metrics and Similarity Measures6.4 (*) KNN Classification Error Bounds6.5 Efficient Data Structures for KNN6.6 The Curse of Dimensionality6.7 Radius-based Nearest Neighbors6.8 K-Nearest Neighbors Regression6.9 Variants of KNN6.10 Applications of KNN6.11. Summary6.12 Exercises
7.1 The Naive Bayes Model7.2 Event Models in Naive Bayes7.3 Naive Bayes Classifiers in Scikit-Learn7.4 (*) Linear and Quadratic Discriminant Analysis7.5 Introduction to Natural Language Processing7.6 Document Classification Example7.7 Bayesian Networks7.8 Summary7.9 Exercises
8.1 Decision Tree Definition8.2 Decision Tree Construction8.3 Tree Pruning8.4 Decision Tree Algorithms8.5 Decision Trees in Scikit-Learn8.6 Regression Trees8.7 Multi-Output Problems8.8 Oblique Decision Trees8.9 Summary8.10 Exercises
9.1 Introduction and Motivation9.2 Types of Ensemble Methods9.3 Using Different Learning Algorithms9.4 Bagging Ensembles9.5 Random Forests9.6 Boosting Ensembles9.7 AdaBoost9.8 Gradient Boosting9.9 Stacking Ensembles9.10 Summary9.11 Exercises
10.1 XGBoost10.2 LightGBM10.3 CatBoost10.4 The Higgs Boson Machine Learning Challenge10.5 Summary10.6 Exercises
11.1 Hard-Margin SVM11.2 Soft-Margin SVM11.3 Kernel Methods11.4 Non-Linear SVM11.5 ν-SVM11.6 Support Vector Regression (SVR)11.7 (*) Efficient Methods for Training SVMs11.8 Model Calibration11.9 Summary11.10 Exercises
12.1 Supervised Learning Summary12.2 Choosing a Learning Algorithm12.3 Conducting Machine Learning Experiments12.4 Research in Machine Learning12.5 Machine Learning Competitions12.6 Additional Resources12.7 Exercises

Content preview from Machine Learning Foundations, Volume 1: Supervised Learning

Chapter 10. Gradient Boosting Libraries

This chapter explores recent variants of gradient boosting: XGBoost, LightGBM, and CatBoost, which introduce several innovative techniques to enhance the performance, scalability, and robustness of gradient boosting. Each of these algorithms is supported by a dedicated library that provides optimized implementations and unique features. These libraries have set new standards for efficiency and accuracy in supervised learning tasks and are capable of handling massive datasets with millions of samples efficiently.

XGBoost is renowned for its scalability and flexibility, making it a top choice for many data science competitions and large-scale applications. LightGBM is known for its rapid training performance, ...