Neural Network Programming with TensorFlow

Auto differentiation

Auto differentiation is also known as algorithmic differentiation, which is an automatic way of numerically computing the derivatives of a function. It is helpful for computing gradients, Jacobians, and Hessians for use in applications such as numerical optimization. Backpropagation algorithm is an implementation of the reverse mode of automatic differentiation for calculating the gradient.

In the following example, using the mnist dataset, we calculate the loss using one of the loss functions. The question is: how do we fit the model to the data?

We can use tf.train.Optimizer and create an optimizer. tf.train.Optimizer.minimize(loss, var_list) adds an optimization operation to the computational graph and automatic differentiation ...