Natural Language Processing with Spark NLP

Chapter 15. Chatbot

When we discussed language models, we showed how we can generate text. Building a chatbot is similar, except that we are modeling an exchange. This can make our requirements more complex or, actually, more simple depending on how we want to approach the problem.

In this chapter we will discuss some of the ways this can be modeled, and then we will build a program that will use a generative model to take and then generate responses. First, let’s talk about what discourse is.

Morphology and syntax tell us how morphemes are combined into words, and words into phrases and sentences. The combination of sentences into larger language acts is not as easily modeled. There is an idea of an inappropriate combination of sentences. Let’s look at some examples:

I went to the doctor, yesterday. It is just a sprained ankle. I went to the doctor, yesterday. Mosquitoes have 47 teeth.

In the first example, the second sentence is obviously related to the first. From these two sentences, combined with common knowledge, we can infer that the speaker went to the doctor for an ankle problem that turned out to be a sprain. The second example makes no sense. From a linguistics point of view, sentences are generated from concepts and then encoded into words and phrases. The concepts that are expressed by sentences are connected, so a sequence of sentences should be connected by similar concepts. This will be true whether there is only one speaker or more in a conversation.

The pragmatics ...