Chapter 6Create and Manage Batch Processing and Pipelines
Having programs run on a reoccurring schedule is a very common scenario, especially when working with data. These programs are usually referred to as batch jobs and target repetitive long‐running tasks or tasks that operate on large datasets. A batch job consists of some source code that connects to a database or accesses a data file, then moves the data or manipulates it in some way. After the transfer and/or manipulation is complete, the output is stored in a location for consumption. In large enterprises a team of people can be responsible for managing ...
Get MCA Microsoft Certified Associate Azure Data Engineer Study Guide now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
