Architecting Modern Data Platforms

Chapter 5. Organizational Challenges

In this chapter, we cover organizational aspects of successful Hadoop integration. It is written for decision makers who build the teams that will deal with big data use cases and operations, or for people who influence these decisions. That said, this chapter may also be of interest to a wider audience, including HR professionals. We will highlight how Hadoop’s architecture results in organizational challenges when building teams that run and use it, and we will give you guidelines on what you require from your larger organization to overcome those challenges and ensure the success of Hadoop.

Although it is mostly apparent to technical staff that shifting an organization to be data-driven means a big paradigm change, Hadoop projects often end up being just another piece of software to onboard, and human factors are omitted. Especially in the case of Hadoop, that can be a mistake. Just sticking to the traditional concepts of IT operations may not result in negative effects when deploying Hadoop at small scale and the PoC level, but as you scale and take your clusters to production, problems are almost guaranteed.

To address these appropriately, you should plan for a new team, one that is dedicated to Hadoop operations. This team will not only combine many of the existing skills in your organization but will also include disciplines that are not yet common in corporate IT.

For this discussion we assume classic on-premises IT operations. As with ...