0

After launching the API dataflow.projects.locations.templates.launch(i have tested also "create" api), with a template generated before on dataprep, a column generated with today() function (i have tested also "now()" function) seems not be evaluated during job execution. The date reported is always filled with the day where the template was created the first time.

Is it a correct behavior? why is the column not populated with the date on which the job is launched?

1 Answers1

0

This is indeed the expected behavior, and this is in fact documented in the "Known Limitations" section on the documentation page about running a Dataprep job on Cloud Dataflow:

Cloud Dataflow templates generated from a Cloud Dataprep by TRIFACTA job are intended for use as a static copy of the job executed at the moment in time.

  • All relative functions are computed based on the moment of execution. Functions such as NOW() and TODAY are not recomputed when the Cloud Dataflow template is executed.
  • To update the output values of these functions, re-run the job in Cloud Dataprep by TRIFACTA through the UI or a scheduled job. Then, execute the Cloud Dataflow template job.

So, as suggested in the documentation snippet above, in order to obtain the current value for the TODAY() / NOW() functions, you will need to re-run the job in Dataprep and execute the template.


SIDE NOTE: If you are interested in understanding why this is the behavior, I recommend you to go through the Dataflow templates documentation, although you can take this just as informative data, given that this does not apply to Dataprep templates (which cannot be customized). A function such as TODAY() or NOW() should be passed as a runtime parameter, given that the value will change on execution; in order to enable the use of runtime parameters, Dataflow offers the ValueProvider interface.

dsesto
  • 7,864
  • 2
  • 33
  • 50