Document analysis using Hadoop and Mahout
In this section, we will take an example of document analysis to illustrate analytics using Hadoop and Mahout. We will be using Pig as the higher-level abstraction for Hadoop MapReduce. We will be calculating the distance between documents using a score called Tf-idf. This distance metric is very popular in the field of information retrieval and text analytics. It is based on the statistics of words occurring in a document.
Tf-idf is used to rank documents based on query terms. It is extensively used in text search scenarios. The distance between the query terms and the document terms determines how close the query is with respect to the document. This distance can be used to rank documents.
For this particular ...
Get Hadoop: Data Processing and Modelling now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
