0

In a U-SQL script invoked by an Azure data factory Pipeline, I have this statement to define an input file for my procedure: DECLARE @inputpattern_csv string = "/FOLDER/" + @year.ToString() + @month.ToString().PadLeft(2, '0') + "{daynum:}_ex1.csv";

The procedure has always worked, but now when I run it, it returns this error: E_CSC_USER_INVALIDFILESETPATTERN: Invalid file set pattern 'daynum:' in 'adl://**************.azuredatalakestore.net/FOLDER/201904{daynum:}_ex1.csv'

And the error output on Azure Data Factory is the following:

{
"errorCode": "2703",
"message": "Error Id: E_CSC_USER_INVALIDFILESETPATTERN, Error Message: Invalid file set pattern 'daynum:' in 'adl://**************.azuredatalakestore.net/FOLDER/201904{daynum:}_ex1.csv'.. ",
"failureType": "UserError",
"target": "U-sql Activity"
}

Is a consequence of the update to Azure Data Lake Storage Gen2? How I can solve it?

Thanks

Lorenzo Benassi
  • 621
  • 1
  • 8
  • 31
  • What do you mean by: "Is a consequence of the update to Azure Data Lake Storage Gen2?" Are you referring to moving from Gen1 to Gen2, or are you referring to a product update? – MartinJaffer-MSFT Jul 01 '19 at 22:49
  • You don't need a colon after `daynum` if you don't specify format. – n0rd Apr 24 '20 at 06:04

1 Answers1

-1

It seem you are define the list file from the specify date range, right ? If it correct, may be the code below will help you.

You can refer to MS's document here

DECLARE @File_Set_Pattern = "/sample/{Virtual_Col_Name:yyyy}/{Virtual_Col_Name:MM}/{Virtual_Col_Name:dd}.csv; 

DECLARE @From_Date = new DateTime(2019,01,01); 
DECLARE @To_Date = new DateTime(2019,01,28); 

@N = EXTRACT 
        Column_Names string, //define list columns and datatype here
        Virtual_Col_Name DateTime from @File_Set_Pattern //this is virtual column name
USING Extractors.Csv(); //replace by extractor you are using

OUTPUT( SELECT * FROM @N WHERE Virtual_Col_Name BETWEEN @From_Date AND @To_Date ) TO "/sample/outputfile.csv USING Outputters.Csv;
Ca Pham Van
  • 316
  • 1
  • 3
  • 12
  • I don't have a range of dates to specify unfortunately! I would like to read all the files contained in a folder and the parameter of the date I must be able to put it every time I run the procedure – Lorenzo Benassi Jun 26 '19 at 07:48