Engineering Optimization

10Gradient‐Based Optimizer Solutions: LM, RLM, CG, BFGS, RG, and GRG

10.1 Introduction

Features of a particular application (constraints, inflections, steep valleys) provide difficulty for SQ and NR, which are second‐order model‐based optimizers. This chapter covers several solutions that typify a myriad of similar adaptations of the basic incremental steepest descent and Newton‐type approaches. The topics of this chapter are Levenberg–Marquardt (LM), conjugate gradient (CG), Broyden–Fletcher–Goldfarb–Shanno (BFGS), and generalized reduced gradient (GRG). LM blends ISD and NR to ensure it moves downhill, not to a saddle, maximum, or jump to absurd locations. CG and BFGS are for unconstrained applications with CG being a variant of Cauchy’s sequential line search method and BFGS a quasi‐Newton approach. GRG is for handling constraints.

Although these are established and intellectually clever solutions, they are limited to certain application classes and add algorithm complexity. Although they are often considered best‐in‐class solutions, I favor the direct search approaches of Chapter 11 for simplicity, generality, and effectiveness.

10.2 Levenberg–Marquardt (LM)

The concept is that for the early iterations, use incremental steepest descent. This ensures that the TS moves in a downhill sequence. As you approach the optimum and are more likely to be in a locally quadratic convex region of the function, switch to Newton–Raphson. The switch will be gradual, not abrupt. As a note ...