Practical Natural Language Processing

Chapter 11. The End-to-End NLP Process

The process is more important than the goal. The person you become is infinitely more valuable than whatever the result is.

Anthony Moore

So far in the book, we’ve addressed a range of NLP problems, starting from what an NLP pipeline looks like to how NLP is applied in different domains. Efficiently applying what we’ve learned to build end-to-end software products involving NLP takes more than just stitching together various steps in an NLP pipeline—there are several decision points during the process. While a lot of this knowledge comes only with experience, we’ve distilled some of our knowledge about the end-to-end NLP process in this chapter to help you hit the ground running faster and better.

In Chapter 2, we already saw what a typical pipeline for an NLP system looks like. How is this chapter then any different from that? In Chapter 2, we focused primarily on the technical aspects of the pipeline—for example, how do we represent text? What pre-processing steps should we do? How do we build a model, and then how do we evaluate it? In the subsequent chapters in Parts I and II of the book, we delved deeper into different algorithms to perform various NLP tasks. We also saw how NLP is used in various industry domains, such as healthcare, e-commerce, and social media. However, in all these chapters, we spent little time on the issues related to deploying and maintaining such systems and on the processes to follow when managing such projects. ...