O'Reilly logo

Deep Learning Cookbook by Douwe Osinga

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 6. Question Matching

We’ve now seen a few examples of how we can construct and use word embeddings to compare terms with one another. It’s natural to ask how we can extend this idea to larger blocks of text. Can we create semantic embeddings of entire sentences or paragraphs? In this chapter, we’ll try to do just that: we’re going to use data from Stack Exchange to build embeddings for entire questions; we can then use those embeddings to find similar documents or questions.

We’ll start out by downloading and parsing our training data from the Internet Archive. Then we’ll briefly explore how Pandas can be helpful for analyzing data. We let Keras do the heavy lifting when it comes to featurizing our data and building a model for the task at hand. We then look into how to feed this model from a Pandas DataFrame and how we can run it to draw conclusions.

The code for this chapter can be found in the following notebook:

06.1 Question matching

6.1 Acquiring Data from Stack Exchange

Problem

You need to access a large set of questions to kick-start your training.

Solution

Use the Internet Archive to retrieve a dump of questions.

A Stack Exchange data dump is freely available on the Internet Archive, which hosts a number of interesting datasets (as well as striving to provide an archive of the entire web). The data is laid out with one ZIP file for each area on Stack Exchange (e.g., travel, sci-fi, etc.). Let’s download the file for the travel section:

xml_7z = utils.get_file ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required