read gen2 secondary account with databricks-CodePudding

I'm trying to read the secondary gen2 account in databricks, but I get the following error.

java.io.FileNotFoundException: Operation failed: "The specified resource does not exist.", 404

follow the way i'm making the connection:

spark.conf.set( "fs.azure.account.key..dfs.core.windows.net", "acess_key")

What is the correct way to log into a gen2 secondary account?

CodePudding user response：

You can't access ADLS Gen2 using storage access key. If you look into documentation, you'll see that only following methods are supported:

Azure Active Directory credential passthrough
OAuth using Service Principals
Shared Access Signature (SAS), but you'll need to provide custom code that will generate that token

Storage access token is supported only for blob storage via wasbs protocol. And although it's possible to access ADLS Gen2 using wasbs, it's not recommended because there are some edge cases when reading & writing data.

CodePudding user response：

You can access gen2 account databricks based on the following configuration .

spark.conf.set(
"fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",
dbutils.secrets.get(scope="<scope-name>",key="<storage-account-access-key-name>"))

You can refer here.

NOTE : Accessing any storage container through access key is highly not recommended.