Hands-On Mathematics for Deep Learning

Simulated annealing

Simulated annealing is inspired by the field of metallurgy, where we use heat to alter the properties of a material. The applied heat increases the energy of ions and moves more freely. As the material starts to cool, it takes on a different shape upon reaching its equilibrium state. The heat needs to be slowly and gradually reduced to avoid the material getting stuck in a metastable state, which represents a local minimum.

In our case, to optimize a problem, we use temperature to control stochasticity. When the temperature is high, this means the process is freely and randomly exploring the space with the hope that it comes across a good convex region with a more favorable minimum. By reducing the temperature, we reduce ...