Deep Learning Cookbook

Chapter 6. Question Matching

We’ve now seen a few examples of how we can construct and use word embeddings to compare terms with one another. It’s natural to ask how we can extend this idea to larger blocks of text. Can we create semantic embeddings of entire sentences or paragraphs? In this chapter, we’ll try to do just that: we’re going to use data from Stack Exchange to build embeddings for entire questions; we can then use those embeddings to find similar documents or questions.

We’ll start out by downloading and parsing our training data from the Internet Archive. Then we’ll briefly explore how Pandas can be helpful for analyzing data. We let Keras do the heavy lifting when it comes to featurizing our data and building a model for the task at hand. We then look into how to feed this model from a Pandas DataFrame and how we can run it to draw conclusions.

The code for this chapter can be found in the following notebook:

06.1 Question matching

6.1 Acquiring Data from Stack Exchange

Problem

You need to access a large set of questions to kick-start your training.

Solution

Use the Internet Archive to retrieve a dump of questions.

A Stack Exchange data dump is freely available on the Internet Archive, which hosts a number of interesting datasets (as well as striving to provide an archive of the entire web). The data is laid out with one ZIP file for each area on Stack Exchange (e.g., travel, sci-fi, etc.). Let’s download the file for the travel section:

xml_7z = utils.get_file ...