Chapter 5. Architecting for Data Reliability
Airbnb, the global online vacation marketplace, wrote in a 2020 post on their engineering blog that âleadership [set] high expectations for data timeliness and quality,â leading to the need to make significant investment in their data quality and governance efforts. Meanwhile, Krishna Puttaswamy and Suresh Srinivas, former engineers at Uber, wrote in a 2021 Uber Engineering blog article that high-quality big data is âat the heart of this massive transformation platform.â
Itâs no secret: data quality is top of mind for some of the best data teams. Still, itâs one thing to write about it: how do we actually achieve this in practice?
Data reliabilityâan organizationâs ability to deliver high data availability and health throughout the entire data life cycleâis the outcome of high data quality. As companies ingest more operational and third-party data than ever before, with employees from across the organization interacting with that data at all stages of its life cycle, itâs become increasingly important for that data to be reliable.
Data reliability has to be intentionally built into every level of your organization, from the processes and technologies you leverage to build and manage your data stack to the way you communicate and triage data issues further downstream. In this chapter, weâll explore how to architect for data reliability at each stage of the pipelineâand data engineering experience.
Measuring and ...
Get Data Quality Fundamentals now with the O’Reilly learning platform.
O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.