AI and ML for Coders in PyTorch

Chapter 5. Introduction to Natural Language Processing

Natural language processing (NLP) is a technique in AI that deals with the understanding of language. It involves programming techniques to create models that can understand language, classify content, and even generate and create new compositions in language. It’s also the underlying foundation to large language models (LLMs) such as GPT, Gemini, and Claude. We’ll explore LLMs in later chapters, but first, we’ll look at more basic NLP over the next few chapters to equip you for what’s to come.

There are also lots of services that use NLP to create applications such as chatbots, but that’s not in the scope of this book—instead, we’ll be looking at the foundations of NLP and how to model language so that you can train neural networks to understand and classify text. In later chapters, you’ll also learn how to use the predictive elements of an ML model to write some poetry. This isn’t just for fun—it’s also a precursor to learning how to use the transformer-based models that underpin generative AI!

We’ll start this chapter by looking at how you can decompose language into numbers and how you can then use those numbers in neural networks.

Encoding Language into Numbers

Ultimately, computers deal in numbers, so to handle language, you need to convert it into numerics in a process called encoding.

You can encode language into numbers in many ways. The most common is to encode by letters, as is done naturally when strings are stored ...