Elegant SciPy

Chapter 7. Function Optimization in SciPy

“What’s new?” is an interesting and broadening eternal question, but one which, if pursued exclusively, results only in an endless parade of trivia and fashion, the silt of tomorrow. I would like, instead, to be concerned with the question “What is best?”, a question which cuts deeply rather than broadly, a question whose answers tend to move the silt downstream.

Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance

When hanging a picture on the wall, it is sometimes difficult to get it straight. You make an adjustment, step back, evaluate the picture’s horizontality, and repeat. This is a process of optimization: we’re changing the orientation of the portrait until it satisfies our demand—that it makes a zero angle with the horizon.

In mathematics, our demand is called a “cost function,” and the orientation of the portrait the “parameter.” In a typical optimization problem, we vary the parameters until the cost function is minimized.

Consider, for example, the shifted parabola, f(x) = (x - 3)². We’d like to find the value of x that minimizes this cost function. We know that this function, with parameter x, has a minimum at 3, because we can calculate the derivative, set it to zero, and see that 2(x - 3) = 0 (i.e., x = 3).

But, if this function were much more complicated (e.g., if the expression had many terms, multiple points of zero derivative, contained nonlinearities, or was dependent on more variables), using manual calculation ...