Stack Overflow
Stack Overflow

Questions tagged [azure-data-lake-gen2]

Ask question related to Azure Data Lake Storage Gen2.

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. Data

669 questions
2
votes
1 answer

In databricks using python, dbutils.fs.mount gives java.lang.NullPointerException: authEndpoint trying to mount using abfss. wasbs works fine

When using db.fs.mount in databricks to connect to azure gen2 data lake a authEndpoint error is received when attempting to connect to "abfss://theDir@theDataLake.blob.core.windows.net/" HOWEVER, connecting to…
Dave Russell
  • 23
  • 6
2
votes
1 answer

Azure blob error :The specified blob does not exist, But Blob is present

I got an error after running sometime my azure function, which is used to read the azure blob storage. Error is ID 0dad768d-36d4-4c1a-85ae-2a5122533b3c fail: Function.processor.User[0] Traceback (most recent call last): File…
galiylama
  • 45
  • 2
  • 7
2
votes
2 answers

Azure Function Python write to Azure DataLake Gen2

I want to write a file to my Azure DataLake Gen2 with an Azure Function and Python. Unfortunately I'm having the following authentication issue: Exception: ClientAuthenticationError: (InvalidAuthenticationInfo) Server failed to authenticate the…
Into Numbers
  • 923
  • 11
  • 19
2
votes
1 answer

how to grab next marker (next_marker) in azure cli command az storage fs file list

Need to calculate size of specific containers and folders at ADLS Gen2. Started with command az storage fs file list. However don't understand how to grab next_marker ? It appears in stdout as warning but not in output of command: WARNING: Next…
Alezis
  • 2,659
  • 3
  • 27
  • 34
2
votes
1 answer

SSMS Back-up and Restore SQL Database to Blob Stopped Working After Switch to Azure Storage Gen2 with AD DS

A couple of months ago, our team followed this Microsoft Tutorial on backing up and restoring databases to a blob storage container, allowing us to keep large databases attached to our SQL Server instance, but backed by blob rather than expensive…
Alain
  • 26,663
  • 20
  • 114
  • 184
2
votes
2 answers

Setting data lake connection in cluster Spark Config for Azure Databricks

I'm trying to simplify notebook creation for developers/data scientists in my Azure Databricks workspace that connects to an Azure Data Lake Gen2 account. Right now, every notebook has this at the top: %scala …
Jason Whitish
  • 1,428
  • 1
  • 23
  • 27
2
votes
1 answer

Trying to open parquet in Synapse - cannot be opened because it does not exist or it is used by another process

I am trying to open a Parquet files that is generated by Stream Analytics and stored in Azure Datalake V2. I have connected datalake and Synapse successfully, but I keep getting…
ruffen
  • 1,695
  • 2
  • 25
  • 51
2
votes
1 answer

AuthenticationException when creating Azure ML Dataset from Azure Data Lake Gen2 Datastore

I have an Azure Data Lake Gen2 with public endpoint and a standard Azure ML instance. I have created both components with my user and I am listed as Contributor. I want to use data from this data lake in Azure ML. I have added the data lake as a…
Stian
  • 1,221
  • 1
  • 19
  • 26
2
votes
1 answer

Backup ADLS gen2

I have datalake & datawarehouse containing about 5-10 TBs of data in Azure ADLS gen2, CSV and Delta formats. ADLS's Performance/Tier=Standard/Hot, replication=GRS, type=StorageV2. What is the best way to backup my ADLS gen2 data? From data…
VB_
  • 45,112
  • 42
  • 145
  • 293
2
votes
0 answers

How to access blob storage using Hadoop SDK and Azure MSI authentication

I try to use org.apache.hadoop.fs.FileSystem.get(Config...); method to get on azure storages. In case of Azure Data Lake Gen2 I use URI like: abfs://mydlfilesystem@mydlaccount.dfs.core.windows.net/my_path and set properties: fs.defaultFS =…
Sergey Shabalov
  • 23
  • 4
2
votes
1 answer

Azure DataLakeServiceClient Python - How to append, How to set Offset and Flush Length?

I want to create and repeatedly append to a csv file using DataLakeServiceClient(azure.storage.filedatalake package). The Inital create/write works as follows. from azure.storage.filedatalake import DataLakeServiceClient datalake_service_client =…
SriramN
  • 432
  • 5
  • 19
2
votes
3 answers

Is it possible to create a SAS token for a directory in DataLake Gen2 storage?

I have an Azure Function that triggers from a directory (namespace) nested within an ADLS Gen 2 storage container. Example: ADLS_AccountName/topLevelContainer/Directory1/Directory2/{name} Unfortunately, the Function requires the Connection string of…
ericOnline
  • 1,586
  • 1
  • 19
  • 54
2
votes
1 answer

SQL data in azure datalake gen2

I read that azure datalake gen 2 can store both structured and unstructured data. Where can we store structure data like SQL data in azure datalake gen2 .If I store it in blob, what will be the extension of that file. Is it .bak. Basically any…
MSTechnie
  • 175
  • 10
2
votes
3 answers

How to check whether the storage account V2 created is having data lake gen2 property or not in Azure?

I'm very new to Azure and would like to know how can i check an existing Storage account V2 available in resource group is having type Data lake Gen2 or not. I know the process to create data lake gen 2 by using the option Hierarchical namespace…
pradeep
  • 53
  • 1
  • 7
2
votes
0 answers

java.io.IOException: Stream is closed! Error in HDInsight with ADLS Gen 2

I had originally posted this on the Microsoft Q&A system at this link. but it doesn't appear to be acknowledged or addressed and thought there might be better feedback here on SO. I am currently using Hail for the pyspark library to perform varying…
EagleByte
  • 143
  • 1
  • 8
1 2 3
…
44 45