Document analysis using Hadoop and Mahout

In this section, we will take an example of document analysis to illustrate analytics using Hadoop and Mahout. We will be using Pig as the higher-level abstraction for Hadoop MapReduce. We will be calculating the distance between documents using a score called Tf-idf. This distance metric is very popular in the field of information retrieval and text analytics. It is based on the statistics of words occurring in a document.

Tf-idf is used to rank documents based on query terms. It is extensively used in text search scenarios. The distance between the query terms and the document terms determines how close the query is with respect to the document. This distance can be used to rank documents.

For this particular ...

Get Mastering Hadoop now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.