There was one key insight that brought neural network research out of stagnation and into the modern era: the choice of a better activation function for neurons. Step functions caused issues with the automated training of networks because tiny changes in the network parameters (the weights and biases) could alternately have either no effect or an abrupt major effect on the network. Obviously, this is not a desired property of a trainable system.
The general approach to automatically training ANNs is to start with the output layer and work backwards. For each example in your training set, you run the network in feed-forward mode (that is, prediction mode) and compare the actual output to the desired output. A good ...
