Chapter 5

Building Reliable MapReduce Apps

WHAT’S IN THIS CHAPTER?

  • Unit testing of MapReduce applications using MRUnit
  • Local testing of MapReduce applications
  • Understanding MapReduce support for debugging
  • Utilizing defensive MapReduce programming

WROX.COM CODE DOWNLOADS FOR THIS CHAPTER

The wrox.com code downloads for this chapter are found at www.wiley.com/go/prohadoopsolutions on the Download Code tab. The code is in the Chapter 5 download and individually named according to the names throughout the chapter.

By now, you should be familiar with MapReduce architecture, application design, and customizing MapReduce execution. This chapter discusses how to build reliable MapReduce code by leveraging unit testing and Hadoop-provided facilities for testing applications. You also learn about different approaches to defensive programming that allow your code to cope with partially corrupted data.

UNIT TESTING MAPREDUCE APPLICATIONS

Bugs in code are a fact of life — the more code you write, the more bugs you create. Even the greatest programmers rarely write bug-free code. That’s why code testing is becoming an integral part of code development, and programmers are moving more and more toward test-driven development (TDD).

NOTE This discussion on MRUnit and its usage for unit testing MapReduce jobs was inspired by Michael Spicuzza, a Nokia colleague.
TEST-DRIVEN DEVELOPMENT
Test-driven development (TDD) is a programming technique that requires you to simultaneously write actual ...

Get Professional Hadoop Solutions now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial