AWS CDK StateMachine BatchSubmitJob with dynamic environment variables

Question

I'm trying to create a statemachine with a BatchSubmitJob in AWS CDK with dynamic environment variables in the BatchContainerOverrides. I was thinking about something like this:

container_overrides = sfn_tasks.BatchContainerOverrides(
    environment={
        "TEST.$": "$.dynamic_from_payload"
    }
)

return sfn_tasks.BatchSubmitJob(self.scope,
                                id="id",
                                job_name="name",
                                job_definition_arn="arn",
                                job_queue_arn="arn",
                                container_overrides=container_overrides,
                                payload=sfn.TaskInput.from_object({
                                    "dynamic_from_payload.$": "$.input.some_variable"
                                }))

However, upon deployment, CDK will add "Name" and "Value" to the statemachine definition, but Value is now static. This is part of the statemachine definition as seen in the console:

"Environment": [
    {
        "Name": "TEST.$",
        "Value": "$.dynamic_from_payload"
    }
]

But I need to have it like this:

"Environment": [
    {
        "Name": "TEST",
        "Value.$": "$.dynamic_from_payload"
    }
]

I also tried using "Ref::", as done here for the command parameters: AWS Step and Batch Dynamic Command. But this doesn't work either.

I also looked into escape hatches, overwriting the CloudFormation template. But I don't think that is applicable here, since the generated statemachine definition string is basically one large string.

I can think of two solutions, both of which don't make me happy: override the statemachine definition string with escape hatches with a copy in which "Value" is replaced on certain conditions (probably with regex) OR put a lambda in the statemachine that will create and trigger the batch job and a lambda that will poll if the job is finished.

Long story short: Does anyone have an idea of how to use dynamic environment variables with a BatchSubmitJob in CDK?

Were you able to find a solution to this? – aberger Oct 19 '21 at 14:58 — aberger, Oct 19 '21 at 14:58

score 1 · Accepted Answer · answered Nov 22 '21 at 14:05

1

You can use the aws_cdk.aws_stepfunctions.JsonPath class:

container_overrides = sfn_tasks.BatchContainerOverrides(
    environment={
        "TEST": sfn.JsonPath.string_at("$.dynamic_from_payload")
    }
)

answered Nov 22 '21 at 14:05

K. Galens

26
2

Thanks! I ended up with a Pass state with intrinsic functions to format the value and the aws_cdk.aws_stepfunctions.JsonPath for the BatchSubmitJob. – Daphne Nov 24 '21 at 08:59

score 1 · Answer 2 · answered Nov 24 '21 at 08:58

Solved thanks to K. Galens!

I ended up with a Pass state with intrinsic functions to format the value and the aws_cdk.aws_stepfunctions.JsonPath for the BatchSubmitJob.

So something like this:

sfn.Pass(scope
    id="id",
    result_path="$.result",
    parameters={"dynamic_from_payload.$": "States.Format('static_sub_part/{}',$.dynamic_sub_part)"} )

...

container_overrides = sfn_tasks.BatchContainerOverrides(
    environment={
        "TEST": sfn.JsonPath.string_at("$.result.dynamic_from_payload")
    }
)

AWS CDK StateMachine BatchSubmitJob with dynamic environment variables

2 Answers2