AI and ML for Coders in PyTorch

Chapter 6. Making Sentiment Programmable by Using Embeddings

In Chapter 5, you saw how to take words and encode them into tokens. Then, you saw how to encode sentences full of words into sequences full of tokens, padding or truncating them as appropriate to end up with a well-shaped set of data that you can use to train a neural network. However, in none of that was there any type of modeling of the meaning of a word. And while it’s true that there’s no absolute numeric encoding that could encapsulate meaning, there are relative ones.

In this chapter, you’ll learn about techniques to encapsulate meaning, and in particular, the concept of embeddings, in which vectors in high-dimensional space are created to represent words. The directions of these vectors can be learned over time, based on the use of the words in the corpus. Then, when you’re given a sentence, you can investigate the directions of the word vectors, sum them up, and from the overall direction of the summation, establish the sentiment of the sentence as a product of its words. Also, related to this, as the model scans the sentences, the positioning of the words in the sentence can also help train an appropriate embedding.

In this chapter, we’ll also explore how that works. Using the News Headlines Dataset for Sarcasm Detection dataset from Chapter 5, you’ll build embeddings to help a model detect sarcasm in a sentence. You’ll also work with some cool visualization tools that help you understand how words in a corpus ...