Mining the Social Web, 3rd Edition

Chapter 5. Mining Text Files: Computing Document Similarity, Extracting Collocations, and More

This chapter introduces some fundamental concepts from text mining¹ and is somewhat of an inflection point in this book. Whereas we started the book with basic frequency analyses of Twitter data and gradually worked up to more sophisticated clustering analyses of messier data from LinkedIn profiles, this chapter begins munging and making sense of textual information in documents by introducing information retrieval theory fundamentals such as TF-IDF, cosine similarity, and collocation detection. Accordingly, its content is a bit more complex than that of the chapters before it, and it may be helpful to have worked through those chapters before picking up here.

Note

Previous editions of this book featured the now defunct Google+ product as the basis of this chapter. Although Google+ is no longer featured as the basis of examples, the core concepts are preserved and introduced in nearly the same way as before. For continuity, the examples in this chapter continue to reflect Tim O’Reilly’s Google+ posts as with previous editions. An archive of these posts is provided with the book’s example code on GitHub.

Wherever possible we won’t reinvent the wheel and implement analysis tools from scratch, but we will take a couple of “deep dives” when particularly foundational topics come up that are essential to an understanding of text mining. The Natural Language Toolkit (NLTK) is a powerful technology ...