Natural Language Processing with Spark NLP

Preface

Why Natural Language Processing Is Important and Difficult

Natural language processing (NLP) is a field of study concerned with processing language data. We will be focusing on text, but natural language audio data is also a part of NLP. Dealing with natural language text data is difficult. The reason it is difficult is that it relies on three fields of study: linguistics, software engineering, and machine learning. It is hard to find the expertise in all three for most NLP-based projects. Fortunately, you don’t need to be a world-class expert in all three fields to make informed decisions about your application. As long as you know some basics, you can use libraries built by experts to accomplish your goals. Consider the advances made in creating efficient algorithms for vector and matrix operations. If the common linear algebra libraries that deep learning libraries use were not available, imagine how much harder it would have been for the deep learning revolution to begin. Even though these libraries mean that we don’t need to implement cache aware matrix multiplication for every new project, we still need to understand the basics of linear algebra and the basics of how the operations are implemented to make the best use of these libraries. I believe the situation is becoming the same for NLP and NLP libraries.

Applications that use natural language (text, spoken, and gestural) will always be different than other applications due to the data they use. The benefit and ...