Neural Network Programming with TensorFlow

Analyzing the effect of activation functions on the feedforward networks accuracy

In the previous example, we used RELU as the activation function. TensorFlow supports multiple activation functions. Let's look at how each of these activation functions affects validation accuracy. We will generate some random values:

x_val = np.linspace(start=-10., stop=10., num=1000)

Then generate the activation output:

 # ReLU activation  y_relu = session.run(tf.nn.relu(x_val))  # ReLU-6 activation  y_relu6 = session.run(tf.nn.relu6(x_val))  # Sigmoid activation  y_sigmoid = session.run(tf.nn.sigmoid(x_val))  # Hyper Tangent activation  y_tanh = session.run(tf.nn.tanh(x_val))  # Softsign activation  y_softsign = session.run(tf.nn.softsign(x_val)) # Softplus activation ...