Keras Deep Learning Cookbook

Optimization with stochastic gradient descent

Stochastic gradient descent (SGD), in contrast to batch gradient descent, performs a parameter update for each training example, x⁽ⁱ⁾ and label y⁽ⁱ⁾:

Θ = Θ - η∇_Θj(Θ, x⁽ⁱ⁾, y⁽ⁱ⁾)