Designing Big Data Platforms

1An Introduction: What's a Modern Big Data Platform

After reading this chapter, you should be able to:

• Define a modern Big Data platform
• Describe expectations from data
• Describe expectations from a platform

This chapter discusses the different aspects of designing Big Data platforms, in order to define what makes a big platform and to set expectations for these platforms.

1.1 Defining Modern Big Data Platform

The key factor in defining Big Data platform is the extent of data. Big Data platforms involve large amounts of data that cannot be processed or stored by a few nodes. Thus, Big Data platform is defined here as an infrastructure layer that can serve and process large amounts of data that require many nodes. The requirements of the workload shape the number of nodes required for the job. For example, some workloads require tens of nodes for a few hours or fewer nodes for days of work. The nature of the workloads depends on the use case.

Organizations use Big Data platforms for business intelligence, data analytics, and data science, among others, because they identify, extract, and forecast information based on the collected data, thus aiding companies to make informed decisions, improve their strategies, and evaluate parts of their business. The more the data recorded in different aspects of business, the better the understanding. The solutions for Big Data processing vary based on the company strategy.

Companies can either use on‐site or cloud‐based solutions for ...