0

Working from Azure Synapse Notebook, I have mounted the ADLS Gen2 folder using LinkedServiceA as per below command,

mssparkutils.fs.mount(
 "abfss://<CONTAINER>@<STORAGENAME>.dfs.core.windows.net/", #ADLS GEN 2 PATH
 "/adlsmount", #Mount Point Name
 { "linkedService" : "<REPLACE LINKED SERVICE NAME>"})

I am trying to access the mount path using the local file system API as below.

Folder structure is like container/team/SALES/BILLING/YEAR.

LinkedServiceA that is used to create the mount is having access to only SALES and its subfolders.

import os

jobId = mssparkutils.env.getJobId()

synfs_bill_path = f'synfs:/{jobId}/adlsmount/team/SALES/BILLING/YEAR' #SYNFS
local_bill_path=  f'/synfs/{jobId}/adlsmount/team/SALES/BILLING/YEAR' #Local File System

mssparkutils.fs.ls(synfs_bill_path) #this is working
bills = os.listdir(local_bill_path) #this is failing with permission denied error

But i am able to list all the parent directories using Local File System API Path using os lib

local_base_path=  f'/synfs/{jobId}/adlsmount/team/SALES/'
bills = os.listdir(local_base_path) #this is working, 
print(bills ) #lists "BILLING" folder

Error Message

PermissionError: [Errno 13] Permission denied: '/synfs/152/adlsmount/team/SALES/BILLING/YEAR'
Traceback (most recent call last):

Spark API using synfs_bill_path is also working. I wanted to process large number of small files in the SALES/BILLING/YEAR to reduce the number of files.(spark read fails with large number of files)

Despicable me
  • 548
  • 1
  • 9
  • 24

1 Answers1

0

I have tried to repro your code in my lab environment and your code works fine without any errors for me.

enter image description here

Permission denied [Errno 13] is mostly seen when you try to access a path without having the necessary permissions. Please make sure the user has all the necessary permissions.

NiharikaMoola-MT
  • 4,700
  • 1
  • 3
  • 15
  • Same mount path i am able to list using ```mssparkutils.fs.ls(synfs_bill_path)``` but not with ```os.listdir(local_bill_path)``` . only difference is the scheme, "synfs:/" is working, "/synfs" is not working. ACL is set at "team/SALES" and all subfolders – Despicable me Jan 20 '22 at 10:29