Stack Overflow
Stack Overflow

Questions tagged [azure-data-lake]

Azure Data Lake Analytics is a suite of three big data services in Microsoft Azure: HDInsight, Data Lake Store, and Data Lake Analytics. These fully managed services make it easy to get started and easy to scale big data jobs written in Hive, Pig, Spark, Storm, and U-SQL.

Azure Data Lake Analytics is a suite of three big data services in Microsoft Azure: HDInsight, Data Lake Store, and Data Lake Analytics. These fully managed services make it easy to get started and easy to scale big data jobs written in, U-SQL, Apache Hive, Pig, Spark, and Storm.

  • HDInsight is a fully managed, monitored and supported Apache Hadoop service, bringing the power of Hadoop clusters to you with a few clicks.
  • Data Lake Store is a cloud scale service designed to store all data for analytics. The Data Lake Store allows for petabyte sized files, and unlimited accounts sizes, surfaced through an HDFS API enabling any Hadoop component to access data. Additionally, date in Data Lake Store is protected via ACL's that can be tied to an OAuth2 based identity, including those from your on-premises Active Directory.
  • Data Lake Analytics is a distributed service built on Apache YARN that dynamically scales on demand while you only pay for the job that is running. Data Lake Analytics also includes U-SQL, a language designed for big data, keeping the familiar declarative syntax of SQL, easily extended with user code authored in C#.

To learn more, check out: https://azure.microsoft.com/en-us/solutions/data-lake/

1870 questions
0
votes
1 answer

Asp.Net read/write to Azure data lake and Azure SQL Server

I like to create web app to upload file, save to azure data lake, read/write to azure SQL Server. I use my Azure AD clientId/secret to access data lake, my Azure SQL Server connection string like:…
weslleywang
  • 569
  • 2
  • 8
  • 21
0
votes
1 answer

Get metadata from Azure Data Lake store gen1

We would like to get the metadata out of the file system. Is there anything like fsImage which stores such a medata information? We used following command: curl -i -X GET -H 'Authorization: Bearer '…
user2813165
  • 31
  • 1
  • 5
0
votes
2 answers

AccessControlException when .Net Client App accessing Azure Data Lake

I am trying to access a Data Lake from a .Net client application using this example I have registered the Client App in AAD Tenant and using the Client Id and Client secret from there (as what i believe is service-to-service authentication.) The…
Gadam
  • 2,674
  • 8
  • 37
  • 56
0
votes
1 answer

How can you set the database and schema in a script in a Visual Studio Azure Data Lake U-SQL project with code-behind

It used to be the case in a Visual Studio 2017 Azure Data Lake U-SQL Project that when running a usql script file (e.g. Script.usql) with a code behind class, you could simply select (via drop down lists) not only the ADLA (Azure Data Lake…
patrickjlee
  • 180
  • 2
  • 12
0
votes
1 answer

how to import the CSV file directly to Azure DataLake instead of local path using the powershell?

I want to copy the $rootfolder value directly to Azure Data Lake store. I could not figure out the correct syntax of importing it to Data Lake instead of Local path "C:/". So i tried with copying from Local to the Data lake and still it is showing…
user10813834
  • 43
  • 1
  • 3
  • 11
0
votes
3 answers

Counting lines in Azure Data Lake

I have some files in Azure Data Lake and I need to count how many lines they have to make sure they are complete. What would be the best way to do it? I am using Python: from azure.datalake.store import core, lib adl_creds =…
pelos
  • 1,744
  • 4
  • 24
  • 34
0
votes
2 answers

Data Factory V2 Error Code 2200 on Data Lake Sink

Im using the following as a workshop https://www.youtube.com/watch? v=IAqJ6nCDtGc I have some sales data in an Azure SQL Database (Rather than an on premise database) And I have a data lake Gen Storage 1 I've successfully set up the Connectors…
Debbie Edwards
  • 1
  • 1
  • 4
0
votes
1 answer

Azure DataLake (ADLS) BulkDownload Bad Request

I am trying to download the file from adls using the BulkDownload method using BulkDownload but I am getting a BAD Request response as below: Error in getting metadata for path cc-…
Sandeep Pal
  • 1
  • 1
0
votes
1 answer

E_SMS_USER_TABLEFILESDELETEDDIRECTLY error in U-SQL

When we are trying to run a normal U-SQL job using azure data factory, which was executing perfectly fine earlier, suddenly failing with the below error for every table being referred in the U-SQL job. E_SMS_USER_TABLEFILESDELETEDDIRECTLY But,…
Venkataraman R
  • 12,181
  • 2
  • 31
  • 58
0
votes
1 answer

Writing output of String manipulation to Azure Data lake Store Item

When I try to write an output of a String manipulation of Get-AzureRmDataLakeStoreItemContent output to a variable and try to pass it in a variable to New-AzureRmDataLakeStoreItem i am getting error "New-AzureRmDataLakeStoreItem : Invalid content…
Naveen Venugopal
  • 35
  • 4
0
votes
1 answer

Remove end of Line Comma from a Azure data lake store File

I need some inputs on file processing in Azure Data lake storage using Power Shell. I have a pipe Delimited input file in my ADLS Gen 1 Account. The File content looks like below 1|2|3|a,b, 3|4|5|d,h, I am able to remove last comma using powershell…
Naveen Venugopal
  • 35
  • 4
0
votes
1 answer

dataFactory V2 - Wildcards

I am trying to move & decompress data from Azure Data Lake Storage Gen1. I have a couple of files with ".tsv.gz" extension, and I want to decompress and move them to a different folder, which is in the same data lake. I've tried to use the wildcard…
FelipePerezR
  • 155
  • 1
  • 2
  • 14
0
votes
1 answer

Unable to Import Data From Source From Azure DataLake with Power BI : Error Message 'Failed to save modifications to the server'

I'm using the guide shown here to to Get Data from Azure Data Lake. https://learn.microsoft.com/en-us/azure/analysis-services/tutorials/aas-lesson-2-get-data Everything appears to be working fine. However, when I 'Import' from Power BI's Power…
peter
  • 53
  • 1
  • 1
  • 6
0
votes
0 answers

Using Azure SQLDW Polybase for data ingestion from ADLS Gen 1 using vnet service endpoints

I am trying to use Polybase in Azure SQL Data Warehouse (SQLDW) to ingest data (persisted in Parquet format from a Hadoop cluster in a VNET) on Azure Data Lake Store (ADLS) Gen 1. The process is working fine but the throughput I am getting is quite…
rh979
  • 657
  • 1
  • 5
  • 13
0
votes
1 answer

Logic Apps(event hub to adls)

I'm currently working a gig to build a logic app for one of our main clients. They(A) currently have Azure Backup data streaming into an event hub. On the directory is a logic app that's collecting the data from A - Event hub. They have asked us to…
Alex
  • 5
  • 3
1 2 3
…
99
100