The Definitive Guide to Azure Data Engineering: Modern ELT, DevOps, and Analytics on the Azure Cloud Platform

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

R. C. L'EsteveThe Definitive Guide to Azure Data Engineeringhttps://doi.org/10.1007/978-1-4842-7182-7_9

9. Capture Pipeline Error Logs in SQL Database

Ron C. L’Esteve¹  

(1)

Chicago, IL, USA

 

In Chapter 8, I discussed a variety of methods for capturing Azure Data Factory pipeline logs and persisting the data either to a SQL table or within Azure Data Lake Storage Gen2. While this process of capturing pipeline log data is valuable when the pipeline activities succeed, this chapter will cover how to capture and persist Azure Data Factory pipeline errors to a SQL table within the ADF_DB that was created in previous chapters. Additionally, we will recap the pipeline ...