Container not found, and we can't create it using anoynomous credentials in Azure databricks

Question

I have below config where I am reading files from my blob storage from databricks. My first config works and it uses SAS key from blob container. and the next is one where I am unable to read with Storage account key. the error I get is

Container in account .blob.core.windows.net not found, and we can't create it using anoynomous credentials,

The config which works:

spark.conf.set("fs.azure.sas.containername.storageaccount.blob.core.windows.net","sas-key")
dbutils.fs.ls("wasbs://containername@storageaccount.blob.core.windows.net/")
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))

The one which do not work is :( and I get above error)

dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = "/mnt/topics/logs",
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))

Can you provide information about the firewall settings that you are using? — Saideep Arikontham, Apr 05 '23 at 05:46
firewall setting on databricks workspace? where can i find that? — ZZZSharePoint, Apr 05 '23 at 07:35
if there is a firewall issue then 1st config shouldnt work as well — ZZZSharePoint, Apr 05 '23 at 07:36
The firewall of storage account. If the access is disabled for all networks, then I can see the same error. — Saideep Arikontham, Apr 05 '23 at 07:42
You might be right. Try creating a private endpoint for storage and add the virtual network to your databricks worksapce and try to use the above operations again if possible and check — Saideep Arikontham, Apr 05 '23 at 07:56
Also, what is the `public access level` of the container that you are trying to access? — Saideep Arikontham, Apr 05 '23 at 08:00
why should I do tht? its not a firewall issue..if there was anything with firewall, nothing would have worked — ZZZSharePoint, Apr 05 '23 at 08:04

score 0 · Accepted Answer · answered Apr 05 '23 at 08:30

0

Figured out the answer.. you need to have same mount point in the file path. in above code replace file_path with below:

file_path ="/mnt/topics/logs"

which is same as provided in mount path

answered Apr 05 '23 at 08:30

ZZZSharePoint

1,163
1
19
54

Container not found, and we can't create it using anoynomous credentials in Azure databricks

1 Answers1