Stack Overflow
Stack Overflow

Questions tagged [azure-data-lake]

Azure Data Lake Analytics is a suite of three big data services in Microsoft Azure: HDInsight, Data Lake Store, and Data Lake Analytics. These fully managed services make it easy to get started and easy to scale big data jobs written in Hive, Pig, Spark, Storm, and U-SQL.

Azure Data Lake Analytics is a suite of three big data services in Microsoft Azure: HDInsight, Data Lake Store, and Data Lake Analytics. These fully managed services make it easy to get started and easy to scale big data jobs written in, U-SQL, Apache Hive, Pig, Spark, and Storm.

  • HDInsight is a fully managed, monitored and supported Apache Hadoop service, bringing the power of Hadoop clusters to you with a few clicks.
  • Data Lake Store is a cloud scale service designed to store all data for analytics. The Data Lake Store allows for petabyte sized files, and unlimited accounts sizes, surfaced through an HDFS API enabling any Hadoop component to access data. Additionally, date in Data Lake Store is protected via ACL's that can be tied to an OAuth2 based identity, including those from your on-premises Active Directory.
  • Data Lake Analytics is a distributed service built on Apache YARN that dynamically scales on demand while you only pay for the job that is running. Data Lake Analytics also includes U-SQL, a language designed for big data, keeping the familiar declarative syntax of SQL, easily extended with user code authored in C#.

To learn more, check out: https://azure.microsoft.com/en-us/solutions/data-lake/

1870 questions
0
votes
1 answer

Visual Studio Code won't compile or submit USQL script

I am using Visual Studio Code (VS Code) to compile and then submit usql scripts on our Azure platform. Something happend and I cannot compile or submit any usql jobs anymore, with following error message: I tried multiple solutions to fix…
HonzaB
  • 7,065
  • 6
  • 31
  • 42
0
votes
1 answer

Azure data factory data lake analytics linked service - failed because the connected party did not properly respond

I have created a azure data factory to perform some USQL activities. while creating a New Linked Service (Azure Data Lake Analytics) getting following error. while searching on this issue, found that developer facing different kind of issues with…
VMK
  • 35
  • 1
  • 6
0
votes
1 answer

Azure Data Lake Storage Gen2 access token generation - "AADSTS65001: The user or administrator has not consented to use the application with ID

I'm trying to generate access and refresh tokens to be able to sign in to the Azure Data Lake Storage Gen2 using external application with allows OAuth. What was done: Created Storage account using…
Sergey Konotop
  • 121
  • 1
  • 13
0
votes
1 answer

How can I decompress my .zip file and store in ADL/Blob storage?

I have a ftp as a source connection where some I have zip file and others are not in compress form. I want to copy the files from ftp,decompress zip files and put all files into azure data lake or azure blob storage wherever it's possible to get…
sam
  • 131
  • 1
  • 5
  • 16
0
votes
1 answer

Move Files from one directory to another using SSIS Package in Azure Data Lake Store Gen1

I am creating SSIS package and planning to use ADF to run it. I am using Azure Data Lake Gen1 as File Store. And as per our process once a file load completed we will move the file from one directory to another into Data Lake. But not able to find…
Prabir
  • 66
  • 8
0
votes
1 answer

ADFV2, VNET, and Linked Services

I have an ADFv2, ADLS Gen2 and SQL Database all configured on a VNET. I'd like to connect from the data factory to the storage account and SQL database however I get a permission error on the data factory (see below). I know I can stand up an…
Murray Foxcroft
  • 12,785
  • 7
  • 58
  • 86
0
votes
1 answer

Azure function to convert encoded json IOT Hub data to csv on azure data lake store

I have simulated devices which is sending messages to IoT Hub blob storage and from there I am copying data(encoded in JSON format) to Azure Data Lake Gen2 by creating a pipeline using Azure Data Factory. How to convert these json output file to CSV…
Lalatendu Mohanty
  • 29
  • 1
  • 6
0
votes
1 answer

How to process the telemetry json messages in Azure data lake Gen2?

I have simulated devices which is sending messages to IoT Hub blob storage and from there I am copying data(encoded in JSON format) to Azure Data Lake Gen2 by creating a pipeline using Azure Data Factory. How to convert these json output file to CSV…
Lalatendu Mohanty
  • 29
  • 1
  • 6
0
votes
1 answer

I am working on a databricks notebook running with ADLSV2 using service principle id but receive the following error after mounting my drive

I am working on a databricks notebook running with ADLSV2 using service priciple id but receive the following error after mounting my drive. StatusCode=403 StatusDescription=This request is not authorized to perform this operation using this…
Gaurav Gangwar
  • 467
  • 3
  • 11
  • 24
0
votes
1 answer

Upload data to the Azure ADLS Gen2 from on-premise using Python or Java

I have an Azure Storage account with Data Lake Gen2. I would like to upload data from on-premise to the Lake Gen2 file systems using Python (or Java). I have found examples on how to interact with File Shares in the Storage account, yet I could not…
AlexGuevara
  • 932
  • 11
  • 28
0
votes
1 answer

how to use substring and cast as time in u-sql

how can we substring a column and cast it as time in u-sql @mergeddata = SELECT DISTINCT a.hta_coinsid.Substring(12, 5).time AS CreatedTime FROM @file1 AS AS;
steve
  • 79
  • 8
0
votes
1 answer

how to pass Body for ADLS Gen-2 as a stream

I'm Using ADLS Gen2 Path-Update API to update file from ADLS which is already created. As a body i can easily pass string which is working fine but the same with Stream is not working. i'm reading local file data and trying to store it into stream…
dev
  • 163
  • 1
  • 11
0
votes
1 answer

Can Azure Data Factory ingest an XML file stored in Azure Data Lake Store?

Is there some way to ingest XML files from the Azure Data Lake Store Gen1? I'm attempting to create a DataSet in Azure Data Factory using Azure Data Lake Store Gen1. However, XML doesn't appear to be a supported file format - only delimited files,…
IncirlikHodja
  • 1
0
votes
1 answer

Issue with Updating file in ADLS Gen2 using Rest-API

I'm using ADLS Gen2 using rest api Path-Update i'm trying to update data into already created created blank file into ADLS. but whenever i'm trying to use the API i'm getting below response from API. {StatusCode: 202, ReasonPhrase: 'Accepted'} but…
dev
  • 163
  • 1
  • 11
0
votes
2 answers

Creating folders in Azure DataLake gen 2 with Azure CLI

Is it possible to create a folder in Azure DataLake gen 2 with Azure CLI? Something like with this code: dls fs create --account my-storage-account --folder --path my-folder Or are there some other tools which could help to automate creation of…
Mirit
  • 33
  • 3
  • 9
1 2 3
…
99 100