Ask question related to Azure Data Lake Storage Gen2.
Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. Data
Questions tagged [azure-data-lake-gen2]
669 questions
0
votes
1 answer
Attribute error while creating scope to access Azure Datalake Gen2 from Databricks
I was trying to setup using the scopes and I am having a few issues. Any help would be appreciated.
I ran the below commands in Databricks CLI
databricks secrets create-scope --scope dnb-dlg2-dbrcks-scp-stg
databricks secrets put --scope…
kranny
- 17
- 5
0
votes
1 answer
Read .nc files from Azure Datalake Gen2 in Azure Databricks
Trying to read .nc (netCDF4) files in Azure Databricks.
Never worked with .nc files
All the required .nc files are in Azure Datalake Gen2
Mounted above files into Databricks at "/mnt/eco_dailyRain"
Can list the content of mount using…
Sreedhar
- 29,307
- 34
- 118
- 188
0
votes
1 answer
OAuth token for REST API call to Azure Data Lake Storage Gen 2 using service principal
I'm working on an application (one of the core microservices) which will call the Azure ADLS Gen 2 to store files (in a filesystem) for further processing by other components.
I'm trying to obtain an OAuth token for the authentication purposes by…
Sergey
- 381
- 6
- 24
0
votes
1 answer
Create Polybase tables from DataBricks
Being new to Datawarehouse, I have a new requirement to create EXTERNAL TABLE from Datalake(GEN1/GEN2) into DWH from Databricks. I used the link to create the below code.
// Set up the Blob storage account access key in the notebook session…
SDR
- 361
- 1
- 4
- 20
0
votes
2 answers
How to create a folder inside container in Azure Data Lake Storage Gen2 with the help of 'azure-storage' Package
I am using Azure Data Lake Gen2 as my storage. I need to create different folder structure before uploading the files to relevant folders.
I am using "Azure-Storage" javascript library. But I am not able to figure out how to create a folder inside a…
Sakti Behera
- 97
- 3
- 9
0
votes
1 answer
MS Access data into Azure Blob
Data is in MS Access and it's in one of the shared drive on the network. I need this data in azure blob storage as CSV files. Can anyone please suggest me how can this be possible?
user2841795
- 375
- 3
- 10
- 25
0
votes
1 answer
Is there any real use of user-assigned managed identity if all my resources are in the same subscription?
I am trying to create an HDInsight cluster in a certain subscription. Now the default storage type that I am selecting is of type ADLS Gen2 and the storage instance exists in the same subscription (the UI here will anyways list only the ADLS Gen2…
Dhiraj
- 3,396
- 4
- 41
- 80
-1
votes
0 answers
How can test the Data Ingestion from D365 CRM to Data Lake?
I'm looking for ideas/existing solutions to effectively test the Data Ingestion from D365 CRM to Data Lake.
I wanted to know if this is possible/good idea to do ?
I have researched on Fluid Test but that doesn't suit my requirement
Sugnick Sen
- 1
- 2
-1
votes
1 answer
How to rerun the failed pipeline in azure data factory(i want only run failed activity)
I am running a pipeline where im looping through all the excel files into azure sql, but i have one failed excel sheet in the pipeline, remaing files are succssfully, i want rerun the failes sheet.
Chinnu
- 5
- 4
-1
votes
0 answers
Failed to merge schema when using CONVERT TO DELTA on a folder with parquet files
I have parquet files stored in ADLS gen2 with such structure:
year/month/day
part_*.snappy.parquet
The files in the folders represent the same dataset, but which schema is changing ("evolving") over time. So, for 2023-01-15 schema of the parquet…
archjkeee
- 13
- 4
-1
votes
2 answers
Need to find a design plan to this problem related to a real time scenarios. Any ideas are appreciated. Thanks
A fleet of 1000 cabs in the warehouse are tracked by multiple fleet managers. Sensors from each cab generates Telematics data & location information. All the telematics data can be captured in to an Azure data lake in two ways.
Data can be streamed…
subham das
- 9
- 1
-1
votes
1 answer
Looping through directories in Python using f-string
I am trying to access 2 different directories in ADLS using for loop & f-string but facing issues in doing so
# Primary storage info
account_name = "accountnamehere" # fill in your primary account name
container_name = "logscontainer" # fill in…
Lopa
- 51
- 6
-1
votes
1 answer
ADF Industries Best Practices
ADF Industries Best Practices with data size wise flow time with number of records
I need an table with Industries Best Practices for each data load
for eg :
data size : 1 GB
number of records : 1 millions
what is the flow time of an…
Raj
- 221
- 1
- 13
-1
votes
1 answer
Using PySpark to read in datalake table and can't parse timestamp column in Synapse Analytics
I can read in the datalake table and print schema but if I try and display data I get the following error. I am working within Synapse Analytics using a PySpark Notebook and Apache Spark Pool.
See error message:
You may get a different result due to…
Arthur Walsh
- 9
- 2
-1
votes
1 answer
How to use SAS url at directory level in ADLS Gen2 to get contents of folder using python
I have sas url at directory level and want to use to read contents of directory instead of using connection string
pratibha
- 1
- 1