Artificial Intelligence By Example

Displaying the details of the activation function through the preactivation process

We often focus on the activation function in a layer. However, when a system fails and we have to debug it, having a peek at the preactivation function sometimes provides effective clues on what is going wrong. To visualize the preactivation output shown in the following graph, click on Wx_plus_b in layer 1, which shows that many events are influencing the training process:

preactivation output

Several notable operations occur with Wx_plus_b:

• The weights and biases are connected to the training process, which is continuously modifying them; this happens even ...