Ask question related to Azure Data Lake Storage Gen2.
Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. Data
Questions tagged [azure-data-lake-gen2]
669 questions
1
vote
1 answer
How can I troubleshoot my Azure Synapse Serverless SQL Pool external table query throwing a 'location does not exist' error?
When I create and query an external table in Azure Synapse Server less SQL Pool it throws the following error:
External table 'EMP_DATA' is not accessible because location does not exist or it is used by another process.
The external table gets…
Imran Saeed
- 13
- 3
1
vote
1 answer
Azure Synapse Pipeline How to execute each file as sql script?
Is it possible to use Azure Synapse Pipeline to execute a set of files as SQL scripts ?
I have a set of files each of which contains the SQL to create say a stored procedure in my Azure Synapse Dedicated SQL pool. I have saved these SQL files in an…
Scott Stensland
- 26,870
- 12
- 93
- 104
1
vote
2 answers
I am getting an permission error with azure AD
On my Storage account my access of Azure AD has been removed
although on IAM i am able to see my Azure AD as a owner
Can you help me on this
user21492560
- 11
- 1
1
vote
1 answer
ErrorCode: 'AuthorizationPermissionMismatch'
I have used service principal in Authentication type while creating Linked services.
but when i try to create a dataset in ADF (i am using adls) i am not able to see my file on container i am getting an error.
ADLS Gen2 failed for forbidden: Storage…
Shobha Sharma
- 41
- 1
1
vote
1 answer
When should you use a mount point in Azure Synapse Analytics?
The documentation of Azure Synapse Analytics mentions two ways read/write data to an Azure Data Lake Storage Gen2 using an Apache Spark pool in Synapse Analytics.
Reading the files directly using the ADLS store path
adls_path =…
jaysc
- 61
- 4
1
vote
1 answer
Synapse Spark: Python logging to log file in Azure Data Lake Storage
I am working in Synapse Spark and building a logger function to handle error logging. I intend to push the logs to an existing log file (data.log) located in AzureDataLakeStorageAccount/Container/Folder/.
In addition to the root logger I have added…
Morpheus273
- 37
- 1
- 6
1
vote
1 answer
Trouble with Azure Synapse: pipeline cannot execute a stored procedure that works in Develop script
I will give some context regarding our inconvenience in Azure Synapse:
We created a Stored Procedure (it creates a VIEW which reads all the
parquet files in a certain folder) on a Develop script, and it ran
successfully.
We also created the VIEW…
Mateo Estrada
- 15
- 3
1
vote
1 answer
how to setup azure storage blob container as SVN repository accessible by TortoiseSVN
I have an azure storage account and blob containers inside an StorageV2 and installed TortoiseSVN on my local windows machine.
What do I have to configure / setup to use the blob-container as a file-repository with TortoiseSVN?
eid
- 537
- 5
- 12
1
vote
1 answer
What is the difference between Azure's "Data Lake Storage Gen2" and "Data Lake Gen2"?
I'm confused by the options available when creating a storage account on Azure and am looking for clarification.
If I create a new "Storage Account" (Standard tier) from the Azure marketplace, I'm offered to activate "Data Lake Storage Gen2" in the…
Krumelur
- 32,180
- 27
- 124
- 263
1
vote
1 answer
Upload large size file more than 2 GB. What will be the best approach?
I want to upload large size file which is more than 2 GB to Azure data lake / blob storage.
I tried with the Azure's cloud blob method PutBlockListAsync.
ref:…
Devanand Dhage
- 35
- 1
- 2
1
vote
1 answer
How to Read Multiple Files from a Folder in ADLS Gen2 using Pyspark in Azure Synapse and use for Processing?
I am Looking for an solution where i want to read all the files from a folder Name as
**'COUNTRIES DETAIL' containing another folder 'YEAR' which is there in a Container in ADLS GEN2 name 'DETAILS'.
. Currently i have 200 files in my sub folder…
BigData Lover
- 135
- 8
1
vote
1 answer
Error in Synapse dataflow while connecting to SAP SLT via SAP CDC Connector
I have the following Setup:
SAP ECC -> SAP LT (SLT) -> SAP CDC Connector (ODP) -> Azure Synapse/Azure Data Lake Storage Gen2 (parquet)
The connection via the SAP CDC connector is working, I see all available tables and when I choose one table, the…
Frank
- 11
- 3
1
vote
2 answers
How to write .csv File in ADLS Using Pyspark
I am reading json file from adls then write it back to ADLS by changing extension to .csv but some random filename is creating in ADLS (writing script in azure synapse)
One _success file and
part-000-***.csv like this some random file name is…
Athiya
- 27
- 7
1
vote
1 answer
Want to all details of blob storage in one script
Script:
$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName "RD-RDIPCloudMigration-AppResources-devtest" -AccountName "rdipstoragegen2").Value[0]
$ctx = New-AzStorageContext -StorageAccountName "rdipstoragegen2" -StorageAccountKey…
Swarnava Sen
- 17
- 4
1
vote
2 answers
How to walk through ADLS folder using python?
I am using the below code snippet to walk through the folders and files in dbfs using python:
for subdir, dirs, files in os.walk("/dbfs/data"):
for file in files:
if re.search(contrast, file):
df =…
user19930511
- 299
- 2
- 15