Hands-On Mathematics for Deep Learning

The secant method

In Newton's method, we calculated the first and second derivatives, but calculating the Hessian in a large problem is not ideal.

Suppose we have a function, , and n = 50. If we take the first derivative of f, with respect to each case of x_i, we get 50 equations. Now, if we calculate the second derivative, we have 2,500 equations, with respect to x_i and x_j, in a matrix. However, because Hessians are symmetric, we only really have to calculate 1,275 second derivatives. This is still a considerably large amount.

The secant method uses the Newton method, but instead of computing the second derivative, it estimates them using ...