7
Managing Batches and Pipelines
In Chapter 6, Developing a Stream Processing Solution, you grasped the foundational elements of building effective Big Data processing systems with an emphasis on the importance of both batch and streaming solutions in the realm of data engineering. You explored the design principles of streaming systems, utilizing Azure services such as Event Hubs, Azure Stream Analytics (ASA), and Spark Streaming, and handled time series data, including concepts such as windowed aggregates, checkpointing, and schema drifts.
In this chapter, you will be focusing on four broad categories: triggering batch jobs, handling failures in batch jobs, managing pipelines, and configuring version control for pipelines. Once you complete ...
Get Azure Data Engineer Associate Certification Guide - Second Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
