U-SQL is a query language designed for Azure Data Lake. It provides a way to mingle SQL keywords with syntactic C# expressions, so that within a single script, a programmer can schematize the data from an unstructured source, use SQL to aggregate the data into the desired form, and then write the output to a file or table.
Questions tagged [u-sql]
771 questions
2
votes
1 answer
VS 2017 - USQL - Parse pipe delimited file
I'm trying to parse (convert single column with pipes to multiple string columns AND also assign variable types to each of the resulting columns) a pipe-delimited file in USQL / Azure using VS 2017.
The file has been loaded into a folder already but…
screechOwl
- 27,310
- 61
- 158
- 267
2
votes
1 answer
Azure / U-SQL - output query to screen
I'm new to U-SQL in Azure (I'm pretty familiar with SQL Server) and can't figure out how to see the output of a query.
All the example's I'm able to find involve writing the results to a file via an outputter.
Do I need to push everything to a…
screechOwl
- 27,310
- 61
- 158
- 267
2
votes
1 answer
Azure Data Lake - Conditions
I would like to use if-else statement to decide at what location I have to export data.
My case is:
I extract several files from the azure blob storage (it's possible that there are no files!!).
I calculate count of records in file set.
If count of…
peterko
- 503
- 1
- 6
- 18
2
votes
1 answer
Identifying newest records in parallel
We're using U-SQL to extract sensor data from a set of .csv files. Each record contains a sensor ID, time of measurement and value, as well as a time for when the record was…
Dan
- 10,480
- 23
- 49
2
votes
1 answer
Deploying USQL project
I am new to data lake analytics and using USQL.
I am currently setting up data factory pipeline which would replace an existing SSIS workflow. The data factory pipeline would essentially
Extract data transactional database into ADLS
Transform raw…
frictionlesspulley
- 11,070
- 14
- 66
- 115
2
votes
0 answers
How to convert UTF-8 string to byte[] in u-sql custom Extractor?
I am converting the string to byte[] using
byte[] text = System.Text.Encoding.UTF8.GetBytes(string_value);
outputrow.Set(i, text);
But the output csv is not having the byte[] column. I think the column is not created for the records.
Is there any…
Arron
- 1,134
- 2
- 13
- 32
2
votes
1 answer
Truncate table partitions dynamically
We have a large U-SQL table containing simple time series data. The table is partitioned per day. Whenever a new batch of data is received, we need to insert new time series data points AND update any previously received data points with a new…
Dan
- 10,480
- 23
- 49
2
votes
2 answers
u-sql issue using gzip and virtual column
I have a strange issue with a u-sql job that process zipped files.
If I run the u-sql on a normal csv file it works fine. But if I gzip the file it doenst work anymore (generating a E_RUNTIME_USER_EXTRACT_ENCODING_ERROR: Encoding error occured…
John
- 79
- 8
2
votes
1 answer
Can you help me with U-SQL Custom Outputter that generated .Avro file
I am looking for a U-SQL Custom Outputter that generated .Avro file. Can you help me in providing with any suggestion or sample?
Kiran Kolli
- 31
- 1
2
votes
2 answers
How to output multiple files dynamically using U-SQL custom Extractors?
I have created a U-SQL query which reads the data from multiple files. Now I have to output the data in multiple files in some other folder. I have the below example,
U-SQL Output in Azure Data Lake
Used the below script,
DECLARE…
Arron
- 1,134
- 2
- 13
- 32
2
votes
1 answer
U-SQL Schema Discovery
The Data Lake approach (according to slide 5 here) is:
Ingest all data - regardless of requirements
Store all data - in native format without schema definition
Do analysis - using engines like Hadoop
But let's say we have loaded up many many…
wBob
- 13,710
- 3
- 20
- 37
2
votes
1 answer
Why can I not register this C# assembly for a U-SQL Project?
I have a U-SQL project that is referring to a C# project. It contains functions that are nothing special. The U-SQL is able to submit to Azure without any problems, but I get this message when I try to register my C# assembly.
My steps are:
Clean…
Kat
- 2,460
- 2
- 36
- 70
2
votes
1 answer
Optimizer internal error while loading data from U-SQL table
Is there a way to get around this error.
"CQO: Internal Error - Optimizer internal error. Assert:
a_drgcidChild->CLength() == UlSafeCLength(popMS->Pdrgcid()) in
rlstreamset.cpp:499"
Facing this issue while loading data from partitioned U-SQL…
Ahsan Abbas
- 155
- 7
2
votes
1 answer
How we can copy any file within Azure Data Lake Store folders
We already have Move-AzureRmDataLakeStoreItemwhich will move files between folders inside Azure datalake. What I am seeking is to copy files within the datalake without effecting the original file.
The possibilities that I know are-
using USQL to…
Abhishek
- 2,482
- 1
- 21
- 32
2
votes
2 answers
How to get Full Path of file in U-SQL query
Is it possible to get file "full path" from following u-sql code, just like file name column?
DECLARE @in = "D:/Sample/Data/{FileName}";
@drivers =
EXTRACT id string,
first_name string,
last_name string,
…
Kishan Gupta
- 586
- 1
- 5
- 18