Machine Learning Quick Reference

Activation function

Activation function is typically introduced in the neural network in order to induce non-linearity. Without non-linearity, a neural network will have little chance to learn non-linearity. But you might question as to why why we need non-linearity in the first place. If we deem every relationship as a linear one, then the model won't be able to do justice to the actual relationship because having a linear relationship is a rarity. If applied linearity, the model's output won't be a generalized one. 

Also, the main purpose of an activation function is to convert an input signal into an output. Let's say if we try to do away with an activation function, it will output a linear result. Linear function is a polynomial of the ...