Designing Big Data Platforms

12An Ideal Platform

After reading this chapter, you should be able to:

• Understand the use of event sourcing for Big Data
• Understand Kappa architecture
• Learn about shift toward data mesh
• Learn about the use of data reservoirs
• Describe data catalog for the platform
• Understand the need for self‐service platform
• Learn about Big Data abstraction
• Learn about design trade‐offs
• Learn about data ethics

Throughout the book, many great technologies and approaches have been reviewed. The summary of what we have talked about before is found in the previous chapter. In this study, the challenge has been coming up with a simple platform while keeping it efficient in terms of development efforts and running costs. Designing such a platform is subject to many constraints such as talent, growth, size, and budget. There is no one‐size‐fits‐all solution for a Big Data platform. A solution that works for an organization might not apply for another one. Therefore, some of the effective patterns for designing a Big Data platform will be studied here. By synthesizing technologies with patterns, a Big Data platform that delivers tremendous value to the organization can be obtained.

It is believed that designing a world‐class Big Data platform starts with early integration to the rest of the services. If the ingestion is lagging, then we have more troubles in working with the data. The load of work increases from cleansing, processing, maintaining, deploying, and so forth. The earlier the Big Data ...