Natural Language Processing with Spark NLP

Chapter 14. Search Engine

In Chapter 13 we discussed how to organize the knowledge we extract from text so that humans and expert systems can utilize it. Most people do not interact with data via graphs, though—especially text data. People generally want to search and retrieve the text. In Chapter 6, we introduced the basic concepts of information retrieval. We learned how to process text as well as how to calculate TF.IDF. In this chapter, we will build an actual search application.

The first thing we need to think about is what problem we are trying to solve. Rather than contrive a specific use case, let’s try and build an application that others can use to solve specific tasks. We want to build a tool that users can use to create a customized search.

We will need our application to do a few different things:

1. Process text data
2. Index the processed text
3. Query the index
4. Label search results to measure and improve the search experience

We used fictional scenarios in Chapters 12 and 13. Let’s see if we can make a tool that will actually be useful to us. We are the users here. This exercise will be useful because it is not uncommon that you will need to build tools for your own use.

Problem Statement and Constraints

1. What is the problem we are trying to solve?

   We want to be able to build custom search engines that we can improve over time. We want this to be as reusable as possible. So we will want to build some abstractions into our application.

2. What constraints are there?

   We ...