Hands-On Deep Learning with R

Surveying activation functions

The activation functions are the last piece of the neural network that we have not covered in depth yet. To review what we know so far, in a neural network, we start with an input, as we would with any machine-learning modeling exercise. This data consists of a dependent target variable that we would like to predict and any number of independent predictor variables that are to be used for this prediction task.

During the training process, the independent variables are weighted and combined in simulated neurons. A bias function is also applied during this step and this constant value is combined with the weighted independent variable values. At this point, an activation function evaluates an aggregation of the ...