Real-World Hadoop

Chapter 4. Decisions That Drive Successful Hadoop Projects

What are the decisions that drive successful Hadoop projects? The answer lies in part in decisions you make as you plan what project to tackle, the way you design your workflow, and how you’ve set up your cluster to begin with. Apache Hadoop is a powerful technology that has enormous potential, and the ways in which it can and will be used are still being discovered. Whether you are a seasoned Hadoop user or a newcomer to the technology, there are some key decisions and strategic approaches to using Hadoop for big data systems that can help ensure your own success with Hadoop and related tools, and we offer some suggestions here that may help with your choices.

The following list is not a comprehensive “how-to” guide, nor is it detailed documentation about Hadoop. Instead, it’s eclectic. We provide technical and strategic tips—some major and some relatively minor or specialized—that are based on what has helped other Hadoop users succeed. Some of these tips will be helpful before you start using Hadoop, and others are intended for more seasoned users, to guide choices as you put Hadoop to work in development and production settings.

Note that the first three tips are particularly important if you are new to Hadoop and NoSQL databases such as HBase or MapR-DB.

Tip #1: Pick One Thing to Do First

If you work with large volumes of data and need scalability and flexibility, you can use Hadoop in a wide variety of ways to reduce costs, ...