Hands-On Mathematics for Deep Learning

Optimization

Optimization is a branch of applied mathematics that has applications in a multitude of fields, such as physics, engineering, economics, and so on, and is of vital importance in developing and training of deep neural networks. In this chapter, a lot of what we covered in previous chapters will be very relevant, particularly linear algebra and calculus.

As we know, deep neural networks are developed on computers and are, therefore, expressed mathematically. More often than not, training deep learning models comes down to finding the correct (or as close to the correct) set of parameters. We will learn more about this as we progress further through this book.

In this chapter, we'll mainly learn about two types of continuous optimization—constrained ...