Is there any way to minimize U-SQL preparation time?

Question

The preparation time on my U-SQL job is approximately 30 seconds. Is it possible to lower that at all?

My code is as follows:

USE DATABASE x;
USE SCHEMA y;

@results = SELECT RowKey
FROM y.tableName
WHERE USQLApplication2.queryHelper.func().Contains(PartitionKey) AND TimestampTicks < new System.DateTime(2016,12,30).Ticks 
      AND TimestampTicks > new System.DateTime(2016,12,29).Ticks;
OUTPUT @results TO "/data/output.csv"
USING Outputters.Csv();

We won't know how to make the changes to your existing code base without seeing your original code. Please post [a minimal example of what needs to change](//stackoverflow.com/help/mcve), and fully explain what needs to be modified. — gunr2171, Feb 06 '17 at 20:58

Michael Rys · Accepted Answer · 2017-02-07T20:53:10.673

The preparation time consist primarily of the compilation, optimization and code-generation of your script. Since U-SQL currently is run in batch mode, the system is optimized for spending time during preparation to make your runtime shorter for large amounts of data.

In your case, based on the provided script, I would think that the call to the user-defined function probably adds to the preparation time. Let me check with the team if inlining the function call could save some time in that phase.

Edit: Based on your question in another thread, you seem to have a highly partitioned table. If that is correct, then the compilation to access the partitions and union them will be adding to the preparation time as well. We have a backlog item to improve that aspect.

Is there any way to minimize U-SQL preparation time?

1 Answers1