U-SQL is a query language designed for Azure Data Lake. It provides a way to mingle SQL keywords with syntactic C# expressions, so that within a single script, a programmer can schematize the data from an unstructured source, use SQL to aggregate the data into the desired form, and then write the output to a file or table.
Questions tagged [u-sql]
771 questions
3
votes
2 answers
How can I log something in USQL UDO?
I have custom extractor, and I'm trying to log some messages from it.
I've tried obvious things like Console.WriteLine, but cannot find where output is. However, I found some system logs in…
arghtype
- 4,376
- 11
- 45
- 60
3
votes
3 answers
U-SQL Paralell reading from SQL Table
I have a scenario in which I am ingesting data from a MS SQL DB into Azure Data Lake using U-SQL. My table is quite big, with over 16 millions records (soon it will be much more). I just do a SELECT a, b, c FROM dbo.myTable;
I realized, however,…
candidson
- 516
- 3
- 18
3
votes
4 answers
Generate dates in a date range using U-SQL
I need to populate a rowset with all the dates between a defined Start date and End Date. If my start date is 19/7/2017 and the end date is 21/7/2017 then the rowset should contain 19/7/2017, 20/7/2017 and 21/7/2017.
I was wondering if there was an…
Absolute Beginner
- 53
- 5
3
votes
1 answer
u-SQL - is it possible to reference a registered assembly from a code-behind file
Registered assemblies can be used in u-SQL script files by using REFERENCE ASSEMBLY.
Is it possible to reference a registered assembly in a c# code behind file?
Liz Nett
- 139
- 2
- 7
3
votes
1 answer
Value too long failure when attempting to convert column data
Scenario
I have a source file that contains blocks of JSON on each new line.
I then have a simple U-SQL extract as follows where [RawString] represents each new line in the file and the [FileName] is defined as a variable from the @SourceFile…
Paul Andrew
- 3,233
- 2
- 17
- 37
3
votes
1 answer
Cannot access Azure Key Vault from Azure Data Lake Analytics
I have an U-SQL script with custom extractor, which access Azure Key Vault to get some credentials.
I followed this tutorial. And I have equivalent code to get token from AD and then to call provided URI for actual credentials:
public static async…
arghtype
- 4,376
- 11
- 45
- 60
3
votes
1 answer
U-SQL job performance
Could you help me with the job performance? I runned it with 10 AUs. And at first part of time they are used almost all. But from the second half of the execution time it uses only 1 AU. I see in the plan a one supervertex consists from only one…
churupaha
- 325
- 2
- 10
3
votes
2 answers
How to merge base and multiple delta structured streams in Azure DataLake
I have a base stream and I have multiple delta streams for different consecutive dates. I want to merge them to get the final stream. How can I accomplish this in Azure Datalake. For example assume these are the stream. I need to merge these streams…
user2580488
- 73
- 10
3
votes
2 answers
U-SQL Split a CSV file to multiple files based on Distinct values in file
I have the Data in Azure Data Lake Store and I am processing the data present there with Azure Data Analytic Job with U-SQL. I have several CSV files which contain spatial data, similar to this:
File_20170301.csv
longtitude| lattitude | date …
FeodorG
- 178
- 2
- 10
3
votes
1 answer
U SQL: direct output to SQL DB
Is there a way to output U-SQL results directly to a SQL DB such as Azure SQL DB? Couldn't find much about that.
Thanks!
candidson
- 516
- 3
- 18
3
votes
2 answers
Does ROWCOUNT hint works for EXTRACT in U-SQL
I want to allocate more vertexes to the extraction job, tried using ROWCOUNT hint, it doesn't seem to work, no matter what value I use for ROWCOUNT, U-SQL always allocate the same number of vertexes.
EXTRACT xxxx
FROM @"Path"
USING new…
lidong
- 556
- 1
- 4
- 20
3
votes
1 answer
u-sql job is very slow, when i add a .NET call
The code performs very fast over 2000 small files (~10-50 Kb) ~ 1 min. Parallelizm = 5.
@arenaData =
EXTRACT col1, col2, col3
FROM @in
USING Extractors.Tsv(quoting : true, skipFirstNRows : 1, nullEscape : "\\N",…
churupaha
- 325
- 2
- 10
3
votes
1 answer
What do priority and parallelism value mean in Azure Data Lakes (Hadoop)?
In other words, what does a parallelism value of 5 and a priority value of 1000 mean?
Justin Borromeo
- 1,201
- 3
- 13
- 26
3
votes
1 answer
Bad Request.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=Block blobs are not supported
I am trying to create an azure data factory by following this article Create, monitor, and manage Azure data factories using Data Factory .NET SDK
Now I am getting following error:
Copy activity encountered a user error at Sink side:…
Kishan Gupta
- 586
- 1
- 5
- 18
3
votes
1 answer
Is it possible to use U-SQL managed tables as output datasets in Azure Data Factory?
I have a small ADF pipeline that copies a series of files from an Azure Storage Account to an Azure Data Lake account. As a final activity in the pipeline I want to run a U-SQL script that uses the copied files as inputs and outputs the result to a…
soderstromOlov
- 384
- 1
- 5
- 11