Deep Learning Essentials

Automatic differentiation

TensorFlow provides a very convenient API that can help us to directly derive the deltas and update the network parameters:

# Define the cost as the square of the errorscost = tf.square(error)# The Gradient Descent Optimizer will do the heavy liftinglearning_rate = 0.01optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)# Define the function we want to approximatedef linear_fun(x):    y = x[:,0] * 2 + x[:,1] * 4 + 1    return y.reshape(y.shape[0],1)# Other variables during learningtrain_batch_size = 100test_batch_size = 50# Normal TensorFlow - initialize values, create a session and run the modelsess = tf.Session() sess.run(tf.initialize_all_variables())for i in range(1000): x_value = np.random.rand(train_batch_size,2) ...