Azure Databricks Cookbook

Chapter 5: Integrating with Azure Key Vault, App Configuration, and Log Analytics

These days, most data processing or data ingestion pipelines read or write data from or to various external sources such as Azure Blob/ADLS Gen-2, SQL, Cosmos, and Synapse. To access these sources, you will need credentials. Storing these credentials, such as a storage account key or SQL password, in a notebook is not an option from a security standpoint and not a recommended approach when deploying these data ingestion or processing pipelines to production. To overcome these problems, Microsoft provides Azure Key Vault, where you can store credentials such as username and password, and access them securely from your notebooks. Apart from Key Vault, we can also ...