Stack Overflow
Stack Overflow

Questions tagged [azure-data-lake-gen2]

Ask question related to Azure Data Lake Storage Gen2.

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. Data

669 questions
2
votes
1 answer

How to get data from a JSON file using Azure functions and C# which is in Blob storage and Storing JSON data as columns in Azure table storage

How to get data from a JSON file (filename.json) using Azure functions and C# which is in Blob storage and Storing JSON data as columns in Azure table storage. My function app will be triggered when a json file has been placed in the path. So…
Surekha Rajendran
  • 21
  • 3
2
votes
1 answer

databricks init script to mount dbfs on adls

I am using a python notebook to mount dbfs on adls , now I want to add this to the init scrip so this can be done during the job cluster start this is the python code I am using how to make this run as the init script please: environment =…
scalacode
  • 1,096
  • 1
  • 16
  • 38
2
votes
1 answer

Fetch dbfs files as a stream dataframe in databricks

I have a problem where I need to create an external table in Databricks for each CSV file that lands into an ADLS gen 2 storage. I thought about a solution when I would get a streaming dataframe from dbutils.fs.ls() output and then call a function…
LeandroHumb
  • 843
  • 8
  • 23
2
votes
1 answer

Bicep isHnsEnabled Cannot be updated

We are in the process of upgrading our ARM Templates to Bicep, one of which is for storage. In the initially ported Bicep file, everything worked fine and then as part of my PR, it was highlighted that I'd left out isHnsEnabled. I then adjusted my…
Raymondo
  • 483
  • 2
  • 6
  • 20
2
votes
1 answer

Mounting ADLS gen2 with AAD passthrough in Azure Databricks with Terraform

I am trying to mount my ADLS gen2 storage containers into DBFS, with Azure Active Directory passthrough, using the Databricks Terraform provider. I'm following the instructions here and here, but I'm getting the following error when Terraform…
Xophmeister
  • 8,884
  • 4
  • 44
  • 87
2
votes
2 answers

Apache Iceberg table format to ADLS / azure data lake

I am trying to find some integration to use iceberg table format on adls /azure data lake to perform crud operations. Is it possible to not use any other computation engine like spark to use it on azure. I think aws s3 supports this usecase. Any…
John
  • 35
  • 5
2
votes
2 answers

Snowflake table as external table in databricks hive metastore

Does anyone know if it is possible to set a snowflake table as an external table in databricks hive metastore? I'm working on a project in which we have some tables in Azure Data Lake Gen 2 and we are managing them from databricks. For being able to…
Cristian Ispan
  • 571
  • 2
  • 5
  • 23
2
votes
2 answers

Azure Synapse: Cannot bulk load because the file could not be opened. Operating system error code 12(The access code is invalid.)

I am using Azure Synapse to query a large number of CSV files with the OPENROWSET command see here. The files are located on a Data Lake gen 2 connected to the Azure Synapse via a managed identity. This is working fine when I am only querying a few…
William Moore
  • 3,844
  • 3
  • 23
  • 41
2
votes
3 answers

Storage Event Trigger does not work when I upload file in Datalake using Azure Function

I have created a storage event trigger to trigger a pipeline in Azure Data Factory. This trigger works when I manually put a file in the datalake. But when the file is uploaded from Azure Function, trigger doesn't work. Below is my function to…
Nikita M.
  • 21
  • 3
2
votes
1 answer

When connecting PowerBI to Azure Data lake storage v2 ( adls v2 ) ; it gives error 404 "The specified container does not exist"

When connecting PowerBI desktop to Azure Data lake storage v2 ( adls v2 ) ; it gives error 404 "The specified container does not exist". Even though container do exist.
Deepak Singhal
  • 10,568
  • 11
  • 59
  • 98
2
votes
2 answers

azure data flow converting boolean to string

I am trying to split large json files into smaller chunks using azure data flow. It splits the file but it changes column type boolean to string in output files. This same data flow will be used for different json files with different schemas…
Akhilesh Jaiswal
  • 227
  • 2
  • 14
2
votes
1 answer

Azure Storage Explorer failing with Access Control Lists

I've set up my Azure data lake gen2 with Access Control List (ACL) access via AAD groups only (no RBAC). The container folder has been granted execute and read permission, as has the relevant sub folder and files. I can confirm access to the ACL…
Jules
  • 23
  • 5
2
votes
1 answer

Create a dynamic Folder Path in SSIS Flexible File Destination

I am trying to dump data into Azure Datalake Gen2 using SSIS. I found about Flexible File Destination Editor but I am stuck at setting the Folder Path dynamically. I have set the variable thisYear and thisMonth but how do I use those variables in my…
rain123
  • 243
  • 4
  • 13
2
votes
1 answer

Can I access ADLS from Azure Databricks using an ADF managed identity?

I have a ADF which has a managed identity able to access the ADLS. Now I need this ADF to call a Databricks notebook and this notebook should access the ADLS as well. Can I do it using the managed identity from ADF? How can I do this?
Ana Lívia
  • 95
  • 2
  • 10
2
votes
1 answer

Unable to mount Azure ADLS Gen 2 on from Community Edition of Databricks : com.databricks.rpc.UnknownRemoteException: Remote exception occurred

I am trying to mount ADLS Gen 2 from my databricks Community Edition, but when I run the following code: test = spark.read.csv("/mnt/lake/RAW/csds.csv", inferSchema=True, header=True) I get the error: com.databricks.rpc.UnknownRemoteException:…
Patterson
  • 1,927
  • 1
  • 19
  • 56
1 2 3
…
44 45