Designing Big Data Platforms

11Putting All Together

After reading this chapter, you should be able to

• Compare Big Data platform design alternatives
• Review Big Data systems and tools
• Discuss various Big Data challenges

The previous chapters have discussed many Big Data platforms' components and modern Big Data platform technology requirements. The technologies used in different settings and the tools and systems addressing Big Data platform demands are explored. This chapter discusses the challenges while designing Big Data platforms.

11.1 Platforms

Designing and building a Big Data platform requires decisions. Working solutions are available and often used by organizations to deal with Big Data. However, these solutions might be incomplete, incoherent, and hard to maintain. An overall vision for the platform would help organizations. Nevertheless, shifting from systems to platform requires effort and dedication from the executive management and other parties involved.

Big Data platforms focus on infrastructure. One of the current key factors is the deployment model, which includes in‐house solutions that require additional maintenance costs due to the provisioning of resources and talent. On the other hand, an infrastructure created by a cloud provider can be used to build the platform. The last alternative is to employ a hybrid infrastructure where some resources and systems are available from the cloud provider. The rest of the infrastructure is an in‐house solution.

The key element is the long‐term ...