In Databricks we can create and access secrets in two different ways:

1. Azure Key vault backed secrete scope
2. Databrick-backed secret scope.

Please refer to my previous blog post for Databrick-backed secret scope. Here we are going to discuss the Azure Key Vault-backed secret scopes.

1. Verify that you have adequate permissions to access the Azure Key Vault.
2. Create a scope in the Databricks.

To create a scope you need to type this URL in the browser. Please note that this URL is not given in the workspace so you need to construct this URL:

https://<your_azure_databricks_url>#secrets/createScope #for example,  https://westus.azuredatabricks.net#secrets/createScope

You need to enter the Scope name and Key Vault DNS and Resource ID.

Key vault info can be found by clicking the properties section of the key vault as shown below.

3. Now verify that the scope is successfully created by using the Databricks CLI command

databricks secrets list-scopes

4. Now note down the Application client ID and Directory ID from the service principal created to access the data lake so you can use the same in the Powershell.

5. Create a secret in the azure key vault so we can access the same.

6. Let’s try to access the Database from the Azure-backed key vault.

Here we are using the secret called mykey which is stored in the key vault by using the scope(mynewscope) created in step 2. We are trying to access the Data Lake from python code below.

configs = {"dfs.adls.oauth2.access.token.provider.type": "ClientCredential",
           "dfs.adls.oauth2.client.id": "Client ID is the application id taken from above screenshot",
           "dfs.adls.oauth2.credential": dbutils.secrets.get(scope = "mynewscope", key = "mysecretkey"),
           "dfs.adls.oauth2.refresh.url": "https://login.microsoftonline.com/Azure AD Directory ID/oauth2/token" 
 }