The Definitive Guide to Azure Data Engineering: Modern ELT, DevOps, and Analytics on the Azure Cloud Platform

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2021

R. C. L'EsteveThe Definitive Guide to Azure Data Engineeringhttps://doi.org/10.1007/978-1-4842-7182-7_13

13. Incrementally Upsert Data

Ron C. L’Esteve¹  

(1)

Chicago, IL, USA

 

If you recall from Chapters 4 through 6, I demonstrated how to do the following:

• Fully load Azure Data Lake Storage Gen2 from a SQL database.

• Fully load Azure Synapse Analytics DW with the Data Lake Storage Gen2 parquet files using Azure Data Factory’s Copy activity.

This ELT process was driven by a metadata approach by using pipeline parameters populated in a SQL table. While this is a useful ELT method, this chapter will focus on advancing this process to address the following:

• Incrementally ...