How can I provide runtime parameter/argument values via a file for CDAP pipleline

Question

How do i add run time parameter/arguments to CDAP pipeline.

We can set parameters in all the Transforms, Source and Target nodes and manually enter the values at run time in the Datafusion environments.

However in a production environments i would want a parameter file to feed the values to the pipeline, how do i achieve this functionality.

In the CDAP documentation i don't find anything useful regarding the run time arguments. I am new to datafusion and am not very familiar with the interface. If this question is already answered a redirect to the thread will also be helpful

Answer 1

When starting a pipeline, you can specify runtime arguments as a JSON map in the request body.

First, your pipeline should contain property lookup macros, which are documented here. As an example, a pipeline json definition could contain the following configuration for the BigQuery Sink plugin (surrounding fields removed for brevity):

  {
    "name": "BigQuerySink",
    "plugin": {
      "properties": {
        "table": "${bqsink_table}"
      }
    },

Then, when starting your pipeline, you can specify the value(s) for the macros, as mentioned here.

In this example, you would send a POST request to v3/namespaces/<namespace-id>/apps/<app-id>/workflows/DataPipelineWorkflow/start, with request body:

{ "bqsink_table": "myTable" }