Artificial Intelligence By Example

Gradient descent and backpropagation

Gradient descent defines two concepts:

• Gradient or derivative measurement of the slope (up or down / how steep)
• Descent or reducing the error level between the present result, relying on the parameters (weights and biases), and the target training dataset

There are several ways to measure whether you are going up or down a slope. Derivatives are the most commonly used mathematical tool for this. Let us say you have 15 steps to go from one floor of a building down to the next floor. At each step, you feel you are going down.

The slope or derivative is the function describing you going down those stairs:

• S = slope of you going down the stairs
• dy = where you are once you have made a step (up, down, or ...