Chapter 5. Streamlining HDFS for big data

 

This chapter covers
  • Understanding how to work with small files
  • Working with compression
  • Choosing the best codec for splittability and performance

 

In the previous chapter we looked at how to work effectively with MapReduce and big data. You might tend to spend more time thinking about MapReduce because of its computational layer, but you shouldn’t deprive HDFS of attention when wrestling with big data, because improvements to HDFS will pay great dividends in terms of performance with our MapReduce jobs.

In view of this, this chapter is dedicated to looking at ways to efficiently store and access big data in HDFS. The first subject I’ll address is how to work with a large number of small files ...

Get Hadoop in Practice now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.