Regression Analysis with R

Gradient Descent

To perform OLS regression using the GD algorithm, we have to minimize the cost function that measures how good a given regression line is.

This cost function will take in an (α , β) pair and return an error value based on how well the line fits our data. To compute this error for a given line, we'll iterate through each (x,y) point in our dataset and sum the square distances between each point's y value and the candidate line's y value (computed at α x + β).

The cost function looks like the following:

If we plot the loss function (z axis) with respect to the coefficients of linear regression (α and β), the 3D plot looks like ...