Language Models in Plain English

Chapter 2. Meet the Neural Model Family

If you have read about neural networks, you are probably aware that they are so called because their general structure is inspired by the observed behavior of biological neurons. However, this is not the same as saying that a neural network constructed to perform machine learning tasks works in the same way as a human brain.

An artificial neural network is intended to mimic certain things the human brain does. But in the same way that eating bugs and using sonar cannot let a human know what it is like to be a bat, learning to predict the next word does not allow an LM to know what it is like to be a human. This goes both ways: the fact that our behavior inspires neural network architecture, and there is convergence in the observed outcome (a word is, indeed, predicted), does not mean that we know what it is like to be an LM.

We therefore introduce neural network language models, and all subsequent techniques in this report, by first making critical observations of how humans approach language, memory, and communication. It is important to develop good intuition about the basic building blocks of a neural network language model so that we may take on the increasingly complex models that are coming to the forefront of language modeling.

To communicate these underlying concepts, we will begin with the very human process of learning to bake through trial and error. In our upcoming example, we make certain assumptions to better understand the ...