2

Good morning all,

I'm looking in Google Data Fusion for a way to make dynamic the name of a source file stored on GCS. The files to be processed are named according to their value date, example: 2020-12-10_data.csv

My need would be to set the filename dynamically so that the pipeline uses the correct file every day (something like this: ${ new Date(). Getfullyear()... }_data.csv

I managed to use the arguments in runtime by specifying the date as a string (2020-12-10) but not with a function.

And more generally is there any documentation on how to enter dynamic parameters with ready-made or custom "functions" (I couldn't find it)

Thanks in advance for your help.

Savannah
  • 103
  • 1
  • 1
  • 5

1 Answers1

1

There is a readymade workaround, you can give a try "BigQuery Execute" plugin.

Steps:

  1. Put below query in SQL

select cast(current_date as string) ||'_data.csv' as filename --for output '2020-12-15_data.csv'

  1. Row As Arguments to 'true'
  2. Now use the above arguments via ${filename} wherever you want to.
narendra solanki
  • 374
  • 2
  • 8
  • Thanks for the idea, I'll try this and tell you if it works as desired. It's still surprising that Fusion did not provide a feature for this. – Savannah Dec 15 '20 at 23:25