Stack Overflow
Stack Overflow

Questions tagged [azure-data-lake]

Azure Data Lake Analytics is a suite of three big data services in Microsoft Azure: HDInsight, Data Lake Store, and Data Lake Analytics. These fully managed services make it easy to get started and easy to scale big data jobs written in Hive, Pig, Spark, Storm, and U-SQL.

Azure Data Lake Analytics is a suite of three big data services in Microsoft Azure: HDInsight, Data Lake Store, and Data Lake Analytics. These fully managed services make it easy to get started and easy to scale big data jobs written in, U-SQL, Apache Hive, Pig, Spark, and Storm.

  • HDInsight is a fully managed, monitored and supported Apache Hadoop service, bringing the power of Hadoop clusters to you with a few clicks.
  • Data Lake Store is a cloud scale service designed to store all data for analytics. The Data Lake Store allows for petabyte sized files, and unlimited accounts sizes, surfaced through an HDFS API enabling any Hadoop component to access data. Additionally, date in Data Lake Store is protected via ACL's that can be tied to an OAuth2 based identity, including those from your on-premises Active Directory.
  • Data Lake Analytics is a distributed service built on Apache YARN that dynamically scales on demand while you only pay for the job that is running. Data Lake Analytics also includes U-SQL, a language designed for big data, keeping the familiar declarative syntax of SQL, easily extended with user code authored in C#.

To learn more, check out: https://azure.microsoft.com/en-us/solutions/data-lake/

1870 questions
0
votes
1 answer

get entities all latest records in u-sql script

how to fetch all employees latest records in U-SQL @employees = SELECT * FROM ( VALUES (1, "Noah", 100, (int?)10000, new DateTime(2012,05,31), "cell:030-0074321,office:030-0076545"), (1, "Noah", 100, (int?)10000, new…
steve
  • 79
  • 8
0
votes
1 answer

How to migrate data from local on-premises HDFS to Azure storage

I want to move the data from my local on-premises HDFS server to my Azure HDinsight cluster. I tried distcp command but it does not understand the data lake storage path.
vish
  • 67
  • 3
  • 10
0
votes
2 answers

Logic Apps Azure Data lake Upload file Action-large files are failing to upload with 413 status code

I am trying to upload a file to Azure Data lake using Azure Data lake Upload File action of Logic Apps. It is working fine for small files about 20 MB. But files with 28 MB or greater are failing with Status code 413- request entity too large. I…
P.K
  • 63
  • 7
0
votes
0 answers

AzCopy timeouts when transfering lot of files

I have two Data lake gen2 storage accounts and I need to transfer about 250GB of data from one to another recursively (the whole file system structure). The file system structure contains lot of files (tens of thousands). I've tried: Downloading…
guderkar
  • 133
  • 1
  • 10
0
votes
2 answers

Need solution to integrate Grafana with Azure data lake

I want to integrate Azure data lake storage with Grafana for visualization of time series data. I need to know what all the tools I can use to make it possible. I used ADF to extract data from csv files stored in data lake and move to a table in…
Sakti Behera
  • 97
  • 3
  • 9
0
votes
1 answer

How to use sub-query in U-SQL?

I am using U-SQL to get offset of date according to range(offset, startdate and enddate is available in another set). I have tried using sub query but it throws error. @partnerships = EXTRACT name string, createdon DateTime?, …
Tejas
  • 107
  • 4
  • 13
0
votes
1 answer

When Will Azure ADLS Gen 2 SDK Be Released?

It seems like the SDKs for Data Lake Storage Gen2 are not available now. Are there other ways / workarounds? This seems like a questions many others also have: https://github.com/MicrosoftDocs/azure-docs/issues/22913 Any news about an SDK for gen2…
Kevin Wei
  • 165
  • 11
0
votes
1 answer

Get files list after azure data factory copy activity

Is there a method that gives me the list of files copied in azure data lake storage after a copy activity in azure data factory? I have to copy data from a datasource and after i have to skip files based on a particular condition. Condition must…
101
  • 62
  • 1
  • 6
0
votes
1 answer

"OAuth2Client:Get Token request failed" error in Kubernetes Pod when attempting to make connection to azure datalake

I have a docker image which creates a connection to Azure Datalake using adlCreds = lib.auth(tenant_id=tenantId, client_secret=application_key, client_id=application_id) in python and access a file. I am executing the docker image in a Kubernetes…
williamr21
  • 147
  • 7
0
votes
1 answer

how to Set File Expiry on file Azure Datalake gen1 file at creation time?

Using Azure Data Analytics to run U-SQL queries to write csvs in Azure Data Factory, how can I get an expiry value set on the files? I have a Azure Service Fabric app with a service that writes json files to azure data lakes. In this write process…
Tdawg90
  • 103
  • 1
  • 10
0
votes
1 answer

U-sql - Error on defining input file pattern

In a U-SQL script invoked by an Azure data factory Pipeline, I have this statement to define an input file for my procedure: DECLARE @inputpattern_csv string = "/FOLDER/" + @year.ToString() + @month.ToString().PadLeft(2, '0') +…
Lorenzo Benassi
  • 621
  • 1
  • 8
  • 31
0
votes
0 answers

DateTime in USQL automatically convert to Unix Timestamp in parquet file

I have a problem with DateTime value generated by U-SQL. I wrote some U-SQL to save data into the parquet file, but all the DateTime column automatically converted to Int64 (Unix Timestamp). I tried to investigate and I found some informations at…
Ca Pham Van
  • 316
  • 1
  • 3
  • 12
0
votes
1 answer

How to Connect ADLS Gen-1 with Azure ML Studio

Want to connect ADLS Gen-1 with AzureML Studio. I try to find out some solution but could not get
Gaurav Gangwar
  • 467
  • 3
  • 11
  • 24
0
votes
0 answers

Upload file to Azure Datalake doens't work from SQL Server Agent job

I have created an executable file using C# which uploads file from local to Azure Data-lake. When I run this executable by double clicking on it, it works. But if I try to run the same executable using SQL Server Agent job I get the below…
Subhojit Shaw
  • 53
  • 5
0
votes
1 answer

Adding custom attribute to SQL DB using dynamic parameter in Azure Data factory

I have a CSV file that I need to ingest in MS SQL DB using ADF V2. Now thats a simple thing to achieve and i am able to do it. But I each file also has a Asset ID associated with it which is dynamically generated you can think of it as a GUID.…
Tarun
  • 517
  • 4
  • 9
  • 24
1 2 3
…
99 100