Applied Machine Learning and AI for Engineers

Chapter 13. Natural Language Processing

It’s not difficult to use Scikit-Learn to build machine learning models that analyze text for sentiment, identify spam, and classify text in other ways. But today, state-of-the-art text classification is most often performed with neural networks. You already know how to build neural networks that accept numbers and images as input. Let’s build on that to learn how to construct deep-learning models that process text—a segment of deep learning known as natural language processing, or NLP for short.

NLP encompasses a variety of activities including text classification, named-entity recognition, keyword extraction, question answering, and language translation. The accuracy of NLP models has improved in recent years for a variety of reasons, not the least of which are newer and better ways of converting words and sentences into dense vector representations that incorporate meaning, and a relatively new neural network architecture called the transformer that can zero in on the most meaningful words and even differentiate between different meanings of the same word.

One element that virtually all neural networks that process text have in common is an embedding layer, which uses word embeddings to transform arrays, or sequences, of scalar values representing words into arrays of floating-point numbers called word vectors. These vectors encode information about the meanings of words and the relationships between them. Output from an embedding layer ...