book

Ensemble Methods for Machine Learning

Name: Ensemble Methods for Machine Learning
Author: Gautam Kunapuli
ISBN: 9781617297137

by Gautam Kunapuli

May 2023

Intermediate to advanced

352 pages

11h 19m

English

Manning Publications

Read now

Unlock full access

inside front cover
Ensemble Methods for Machine Learning
Copyright
dedication
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1 The basics of ensembles
1 Ensemble methods: Hype or hallelujah?
1.1 Ensemble methods: The wisdom of the crowds1.2 Why you should care about ensemble learning1.3 Fit vs. complexity in individual models1.3.1 Regression with decision trees1.3.2 Regression with support vector machines1.4 Our first ensemble1.5 Terminology and taxonomy for ensemble methodsSummary
Part 2 Essential ensemble methods
2 Homogeneous parallel ensembles: Bagging and random forests
2.1 Parallel ensembles2.2 Bagging: Bootstrap aggregating2.2.1 Intuition: Resampling and model aggregation2.2.2 Implementing bagging2.2.3 Bagging with scikit-learn2.2.4 Faster training with parallelization2.3 Random forests2.3.1 Randomized decision trees2.3.2 Random forests with scikit-learn2.3.3 Feature importances2.4 More homogeneous parallel ensembles2.4.1 Pasting2.4.2 Random subspaces and random patches2.4.3 Extra Trees2.5 Case study: Breast cancer diagnosis2.5.1 Loading and preprocessing2.5.2 Bagging, random forests, and Extra Trees2.5.3 Feature importances with random forestsSummary

3 Heterogeneous parallel ensembles: Combining strong learners
3.1 Base estimators for heterogeneous ensembles3.1.1 Fitting base estimators3.1.2 Individual predictions of base estimators3.2 Combining predictions by weighting3.2.1 Majority vote3.2.2 Accuracy weighting3.2.3 Entropy weighting3.2.4 Dempster-Shafer combination3.3 Combining predictions by meta-learning3.3.1 Stacking3.3.2 Stacking with cross validation3.4 Case study: Sentiment analysis3.4.1 Preprocessing3.4.2 Dimensionality reduction3.4.3 Blending classifiersSummary
4 Sequential ensembles: Adaptive boosting
4.1 Sequential ensembles of weak learners4.2 AdaBoost: Adaptive boosting4.2.1 Intuition: Learning with weighted examples4.2.2 Implementing AdaBoost4.2.3 AdaBoost with scikit-learn4.3 AdaBoost in practice4.3.1 Learning rate4.3.2 Early stopping and pruning4.4 Case study: Handwritten digit classification4.4.1 Dimensionality reduction with t-SNE4.4.2 Boosting4.5 LogitBoost: Boosting with the logistic loss4.5.1 Logistic vs. exponential loss functions4.5.2 Regression as a weak learning algorithm for classification4.5.3 Implementing LogitBoostSummary
5 Sequential ensembles: Gradient boosting
5.1 Gradient descent for minimization5.1.1 Gradient descent with an illustrative example5.1.2 Gradient descent over loss functions for training5.2 Gradient boosting: Gradient descent + boosting5.2.1 Intuition: Learning with residuals5.2.2 Implementing gradient boosting5.2.3 Gradient boosting with scikit-learn5.2.4 Histogram-based gradient boosting5.3 LightGBM: A framework for gradient boosting5.3.1 What makes LightGBM “light”?5.3.2 Gradient boosting with LightGBM5.4 LightGBM in practice5.4.1 Learning rate5.4.2 Early stopping5.4.3 Custom loss functions5.5 Case study: Document retrieval5.5.1 The LETOR data set5.5.2 Document retrieval with LightGBMSummary
6 Sequential ensembles: Newton boosting
6.1 Newton’s method for minimization6.1.1 Newton’s method with an illustrative example6.1.2 Newton’s descent over loss functions for training6.2 Newton boosting: Newton’s method + boosting6.2.1 Intuition: Learning with weighted residuals6.2.2 Intuition: Learning with regularized loss functions6.2.3 Implementing Newton boosting6.3 XGBoost: A framework for Newton boosting6.3.1 What makes XGBoost “extreme”?6.3.2 Newton boosting with XGBoost6.4 XGBoost in practice6.4.1 Learning rate6.4.2 Early stopping6.5 Case study redux: Document retrieval6.5.1 The LETOR data set6.5.2 Document retrieval with XGBoostSummary
Part 3 Ensembles in the wild: Adapting ensemble methods to your data
7 Learning with continuous and count labels
7.1 A brief review of regression7.1.1 Linear regression for continuous labels7.1.2 Poisson regression for count labels7.1.3 Logistic regression for classification labels7.1.4 Generalized linear models7.1.5 Nonlinear regression7.2 Parallel ensembles for regression7.2.1 Random forests and Extra Trees7.2.2 Combining regression models7.2.3 Stacking regression models7.3 Sequential ensembles for regression7.3.1 Loss and likelihood functions for regression7.3.2 Gradient boosting with LightGBM and XGBoost7.4 Case study: Demand forecasting7.4.1 The UCI Bike Sharing data set7.4.2 GLMs and stacking7.4.3 Random forest and Extra Trees7.4.4 XGBoost and LightGBMSummary
8 Learning with categorical features
8.1 Encoding categorical features8.1.1 Types of categorical features8.1.2 Ordinal and one-hot encoding8.1.3 Encoding with target statistics8.1.4 The category_encoders package8.2 CatBoost: A framework for ordered boosting8.2.1 Ordered target statistics and ordered boosting8.2.2 Oblivious decision trees8.2.3 CatBoost in practice8.3 Case study: Income prediction8.3.1 Adult Data Set8.3.2 Creating preprocessing and modeling pipelines8.3.3 Category encoding and ensembling8.3.4 Ordered encoding and boosting with CatBoost8.4 Encoding high-cardinality string featuresSummary
9 Explaining your ensembles
9.1 What is interpretability?9.1.1 Black-box vs. glass-box models9.1.2 Decision trees (and decision rules)9.1.3 Generalized linear models9.2 Case study: Data-driven marketing9.2.1 Bank Marketing data set9.2.2 Training ensembles9.2.3 Feature importances in tree ensembles9.3 Black-box methods for global explainability9.3.1 Permutation feature importance9.3.2 Partial dependence plots9.3.3 Global surrogate models9.4 Black-box methods for local explainability9.4.1 Local surrogate models with LIME9.4.2 Local interpretability with SHAP9.5 Glass-box ensembles: Training for interpretability9.5.1 Explainable boosting machines9.5.2 EBMs in practiceSummary
epilogue
E.1 Further readingE.1.1 Practical ensemble methodsE.1.2 Theory and foundations of ensemble methodsE.2 A few more advanced topicsE.2.1 Ensemble methods for statistical relational learningE.2.2 Ensemble methods for deep learningE.3 Thank You!
index
inside back cover

Content preview from Ensemble Methods for Machine Learning

5 Sequential ensembles: Gradient boosting

This chapter covers

Using gradient descent to optimize loss functions for training models
Implementing gradient boosting
Training histogram gradient-boosting models efficiently
Gradient boosting with the LightGBM framework
Avoiding overfitting with LightGBM
Using custom loss function with LightGBM

The previous chapter introduced boosting, where we train weak learners sequentially and “boost” them into a strong ensemble model. An important sequential ensemble method introduced in chapter 4 is adaptive boosting (AdaBoost).

AdaBoost is a foundational boosting model that trains a new weak learner to fix the misclassifications of the previous weak learner. It does this by maintaining and adaptively updating ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617297137Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Ensemble Methods for Machine Learning

by Gautam Kunapuli

5 Sequential ensembles: Gradient boosting

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.