Learning TensorFlow

Chapter 6. Text II: Word Vectors, Advanced RNN, and Embedding Visualization

In this chapter, we go deeper into important topics discussed in Chapter 5 regarding working with text sequences. We first show how to train word vectors by using an unsupervised method known as word2vec, and how to visualize embeddings interactively with TensorBoard. We then use pretrained word vectors, trained on massive amounts of public data, in a supervised text-classification task, and also introduce more-advanced RNN components that are frequently used in state-of-the-art systems.

Introduction to Word Embeddings

In Chapter 5 we introduced RNN models and working with text sequences in TensorFlow. As part of the supervised model training, we also trained word vectors—mapping from word IDs to lower-dimensional continuous vectors. The reasoning for this was to enable a scalable representation that can be fed into an RNN layer. But there are deeper reasons for the use of word vectors, which we discuss next.

Consider the sentence appearing in Figure 6-1: “Our company provides smart agriculture solutions for farms, with advanced AI, deep-learning.” This sentence may be taken from, say, a tweet promoting a company. As data scientists or engineers, we now may wish to process it as part of an advanced machine intelligence system, that sifts through tweets and automatically detects informative content (e.g., public sentiment).

In one of the major traditional natural language processing (NLP) approaches to ...