Hadoop: Data Processing and Modelling

Unit testing the Map Reduce code using MRUnit

In this recipe, we are going to learn how to unit test the map reduce code using a library called MRUnit.

Getting ready

To perform this recipe, you should have a running Hadoop cluster as well as an eclipse similar to an IDE.

How to do it...

Sometimes, it is very difficult to develop, compile, deploy and execute the map reduce program and then figure out whether the code is correct or not. In order to avoid this deploying and testing technique, we can use a unit testing framework that's been built specially for Map Reduce called, MRUnit.

Let's assume that we have a map reduce program that emits and counts words starting with '#', such as a Twitter hashtag counter. The Mapper code for this hashtag counter ...