Statistics for Machine Learning

Limited-memory broyden-fletcher-goldfarb-shanno - L-BFGS optimization algorithm

L-BFGS is limited memory of BFGS, which is in the family of quasi-Newton methods that approximate the BFGS algorithm, which utilizes a limited amount of computer memory. BFGS is currently considered the most effective, and is by far the most popular, quasi-Newton update formula.

The L-BFGS methodology is best explained with the following diagram, in which iterations start at a random point (xt) and a second derivative, or hessian matrix, is computed at that point, which is a quadratic approximation of the original function; after computing the quadratic function, it computes the minimum in one step, and after calculating the new point (xt+1) for which the function ...