If the machine learning algorithm is a car, then the optimization algorithm is its engine. For more information, refer to http://www.pyimagesearch.com/2016/10/10/gradient-descent-with-python/.
From your math classes, you should remember that geometrical interpretation is a derivative 
of a function f(x) is a slope at any given point (x) of the function. Now, if we have a function of two parameters f(x, y), we can't calculate just derivative as we did previously. Nevertheless, we can calculate partial derivatives: . The vector composed of these partial derivatives is known as a gradient, and the ...
