Programming Machine Learning

What You Just Learned

Since I introduced gradient descent, we’ve been training all of our ML systems the same way: for each step of gradient descent, we calculated the gradient of the loss over the entire training set. That flavor of gradient descent is called batch gradient descent.

In this chapter, I introduced a different way to do gradient descent: mini-batch gradient descent. In mini-batch GD, we take the loss over a subset of examples at each step. We also tried an extreme variant of mini-batch GD: stochastic gradient descent, where we take the loss on a single example at a time.

Mini-batch GD often converges faster than batch GD. As a result, it gives us early feedback on the training. It’s also perfect for large training sets that don’t ...