As our data pipelines grow in complexity, we need to consider how to productionize them to ensure reliability, scalability, and maintainability. This is where building production pipelines comes in, and it’s the focus of the following pages. In this chapter, we’ll explore how to create robust and efficient production pipelines using Delta Live Tables and Databricks Jobs. We will delve into the nuances of controlling data quality, capturing data changes, and orchestrating workflows to automate our pipelines.
Exploring Delta Live Tables
Delta Live Tables (DLT) is a powerful tool that enables you to build production data pipelines with ease. By providing a simple and intuitive way to manage data pipelines, DLT empowers you to focus on extracting insights from your data. In this section, we will delve into the world of Delta Live Tables, exploring its key features, benefits, and use cases.
Introducing Delta Live Tables
Delta Live Tables is a declarative ETL framework powered by Apache Spark for building reliable and maintainable data pipelines. It’s designed to simplify the process of creating large-scale data processing pipelines, while maintaining table dependencies and data quality.
Figure 6-1 illustrates a sample DLT pipeline, which will be built in the subsequent section of this chapter.
Figure 6-1. Example of a DLT pipeline
As shown, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
