Applied Text Analysis with Python

Chapter 11. Scaling Text Analytics with Multiprocessing and Spark

In the context of language-aware data products, text corpora are not static fixtures, but instead living datasets that constantly grow and change. Take, for instance, a question-and-answer system; in our view this is not only an application that provides answers, but one that collects questions. This means even a relatively modest corpus of questions could quickly grow into a deep asset, capable of training the application to learn better responses in the future.

Unfortunately, text processing techniques are expensive both in terms of space (memory and disk) and time (computational benchmarks). Therefore, as corpora grow, text analysis requires increasingly more computational resources. Perhaps you’ve even experienced how long processing takes on the corpora you’re experimenting on while working through this book! The primary solution to deal with the challenges of large and growing datasets is to employ multiple computational resources (processors, disks, memory) to distribute the workload. When many resources work on different parts of computation simultaneously we say that they are operating in parallel.

Parallelism (parallel or distributed computation) has two primary forms. Task parallelism means that different, independent operations run simultaneously on the same data. Data parallelism implies that the same operation is being applied to many different inputs simultaneously. Both task and data parallelism ...