0

QS 1: How to read a configuration table/file which contain basic properties of a nodes {ex: source and sink table names etc.} and use that output in the next nodes in FUSION pipeline.

tried to use the Remote Program Executor, with command as - 

bq query --format csv 'select SOURCE_TABLE,TASK_ID,DELTA_DATE from FUSIONDATASET.TEST_CONFIG where RUN_STATUS=0'

Now how to use this output in the nest pipeline nodes,say the SOURCE_TABLE col, in the source bigQuery node's TABLE property

user7682941
  • 1

1 Answers1

0

You can use an action plugin to set runtime arguments that can be used in any subsequent stages in a pipeline.

As an example you can refer to argument setter plugin that reads a set of configurations and sets these are runtime arguments. https://github.com/data-integrations/argument-setter/

Specifically the line here https://github.com/data-integrations/argument-setter/blob/develop/src/main/java/io/cdap/plugin/ArgumentSetter.java#L77 sets up key and value of the runtime argument.

Sree
  • 714
  • 4
  • 8