Streaming Data Mesh

Chapter 8. Building a Decentralized Data Team

Companies require a sound strategy in order to shift from an on-premises to a cloud-based architecture. Most companies have seen roles mature and become more specialized, particularly as they relate to data science, data engineering, and machine learning engineering. All of these specialties play a key role in working with data within the cloud ecosystem.

Additionally, the demand for data professionals has risen dramatically over the past few years. This, paired with a steady rate decline of new professionals who are educated in data infrastructure, creates a talent gap where there are not enough professionals to fill the demand for these roles. This gap has created an ongoing demand for software within the data infrastructure space (covered in Chapter 7) that can help with automating key tasks to ease the burden of filling the talent shortage.

Technology is capable of handling some of the existing gaps, but not all. As a result, businesses need to find alternatives to boosting productivity with existing skill sets. As these skill sets are in high demand, additional hiring and staff augmentation are often not an option. Building a decentralized data team helps fill this gap by moving resources within the organization to supply the skill sets necessary to support a streaming data mesh approach.

In this chapter we will review the traditional approach to data, some of its pitfalls, and contrast this with a new approach to aligning resources ...