Machine Learning and AI with Simple Python and Matlab Scripts

10Recurrent Neural Networks

10.1 Introduction

There are many real‐life systems whose outputs depend on the sequence in which the inputs are applied to them. For example, in understanding natural languages, one has to consider multiple words to make sense of a given sentence (e.g. subject, verb, prepositions, adverbs and adjectives), as opposed to analysing each word independently. The next state of such a system depends not only on the current input applied to it but also on the previous inputs. Such a recurrent system possesses the ability to utilize the knowledge that it previously generated, be its previous outputs or other relevant information. The ANNs that we studied in Chapters 2 to 6 consider each input as being independent of the previously applied inputs and, therefore, are not suitable to handle a sequence of inputs.

A recurrent neural network (RNN) is a type of artificial neural network, but with a significant distinction that its intermediate state information and/or outputs are fed as inputs into other neurons (hence the word recurrent in the name). Similar to other neural networks such as ANNs or CNNs, RNNs are also trained by existing field data. However, while the inputs and outputs of ANNs and CNNs are supposed to be independent for each sample, RNN outputs depend on previously applied inputs, their outputs and some of the information created in the process (e.g. so‐called hidden states). In Figure 10.1, two versions of an RNN are shown: compact and unfolded ...