Programming Machine Learning

What You Just Learned

In this chapter we investigated gradient descent, the most widely used algorithm to miminize loss. No matter how complicated our model and dataset are, gradient descent always works the same way: it takes a step in the opposite direction as the gradient of the loss, and keeps doing that until the gradient becomes small. To find the gradient, we took the partial derivatives of the loss with respect to w and b.

GD has a few limitations. In particular, being based on derivatives, it expects that the loss function is smooth and without gaps, so that you can calculate its derivative anywhere. Also, GD can get stuck in a local minimum, failing to reach the global minimum. To avoid those problems, we’ll try to use smooth loss functions ...