Machine Learning in Geomechanics 1

2Introduction to Regression Methods

Filippo MASI

Sydney Centre in Geomechanics and Mining Materials, School of Civil Engineering, University of Sydney, Australia

Regression is one of the fundamental pillars of supervised machine learning (ML). In this chapter, we uncover the essential concepts in regression analysis and methods, by providing (hands-on) practical examples, designed for graduate students and researchers seeking to gain a solid understanding.

In particular, we first delve into linear regression methods, exploring both closed-form solutions and the general optimization framework provided by the gradient descent (GD). Then, we introduce the necessary pre- and post-processing steps for building and testing models: feature scaling, hold-out and cross-validation.

Moving on, we focus on nonlinear regression methods and, in particular, polynomial regression. Through extensive examples, we then demonstrate the benefits associated with regularization techniques (LASSO, Ridge and Elastic Net) and the possibility of constructing interpretable and parsimonious models.

Lastly, we introduce Bayesian approaches with focus on linear and Gaussian process (GP) regression. The advantages and drawbacks of the latter are compared with ordinary regression methods.

2.1. Introduction

Regression is a statistical technique used to explore and quantify the relationship between dependent variables (often called outcomes) and independent variables (often called predictors). In other words, ...