How to create Azure Key Vault-backed secret scope?

In Databricks we can create and access secrets in two different ways:

  1. Azure Key vault backed secrete scope
  2. Databrick backed secret scope.

Please refer to my previous blog post for Databrick backed secret scope. Here we are going to discuss the Azure Key Vault backed secret scopes.

1.Verify that you have the adequate permissions to access the Azure Key Vault.

2. Create a scope in the Databricks.

To create a scope you need to type this url in the browser. Please note that this url is not given in workspace so you need to construct this url:

https://<your_azure_databricks_url>#secrets/createScope #for example,  https://westus.azuredatabricks.net#secrets/createScope

You need to enter the Scope name and Key Vault DNS and Resource ID.

Key Vault specific info can be found by clicking the properties section of the key vault as shown below.

3. Now verify that the scope is successfully created by using Databricks CLI command

databricks secrets list-scopes

4.Now note down the Application client ID and Directory ID from the service principal created to access the data lake so you can use the same in the Powershell.

5.Create a secret in the azure key vault so we can access the same.

6. Let’s try to access the Datalake from Azure backed key vault .

Here we are using the secret called mykey which is stored in the keyvault by using the scope(mynewscope) created at step 2.We are trying to access the datalake from python code below.

configs = {"dfs.adls.oauth2.access.token.provider.type": "ClientCredential",
           "dfs.adls.oauth2.client.id": "Client ID is the application id taken from above screenshot",
           "dfs.adls.oauth2.credential": dbutils.secrets.get(scope = "mynewscope", key = "mysecretkey"),
           "dfs.adls.oauth2.refresh.url": "https://login.microsoftonline.com/Azure AD Directory ID/oauth2/token" 
 }

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.