0

Looking into AWS Glue Workflow for Automation of an ETL pipeline process.

I have defined some workflow parameters to define which customer to run this job for and would like to pass this as input to the Workflow.

Workflow Properties Screenshot

I am confused as to how I can override these default workflow parameters whilst starting a workflow execution from either the AWS CLI

aws glue start-workflow-run --name <Workflow Name>

Or via the AWS Java SDK.

final StartWorkflowRunRequest startWorkflowRunRequest = new StartWorkflowRunRequest();
startWorkflowRunRequest.withName("Workflow Name");

final StartWorkflowRunResult startWorkflowRunResult = awsGlue.startWorkflowRun(startWorkflowRunRequest);
final String runId = startWorkflowRunResult.getRunId();

I see that there is a aws glue put-workflow-run-properties where you can update these values once you have a workflow --run-id executing but hoping to set these at workflow trigger time?

Any help would be greatly appreciated.

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Patrick Bray
  • 552
  • 2
  • 7
  • 20
  • Seems like this may not be possible? If so thinking could move to a Step Functions based approach. – Patrick Bray Nov 10 '21 at 03:07
  • 1
    Old question, but just in case this helps anyone, [it's now possible to pass --run-properties when starting a workflow run](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/start-workflow-run.html). – alscu Mar 09 '22 at 20:23

0 Answers0