6

In Re:Invent 2018, AWS seems to have launched new integrations with Step Functions, which includes ECS Fargate support.

https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ecs.html

I have been digging up on this, and I am able to get the ECS task to startup, and do its thing, but I am not able to send desired output back to the step function. The response syntax too does not seem to have anything that corresponds to the output of a particular state.

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html#API_RunTask_ResponseSyntax

Is it possible for an ECS task to send back output to the step function? Any leads would help.

P.S: I am aware of activities, and have ECS poll for activity, and send task success on completion. But, I was looking to leverage step function calling ECS if that works.

Chandrasekhar Malladi
  • 311
  • 1
  • 3
  • 10

2 Answers2

3

You can use the "Wait for Callback (.waitForTaskToken)" integration pattern to callback to Step Functions in the ECS task. Using .waitForTaskToken instead of .sync, the execution will pause and wait for SendTaskSuccess or SendTaskFailure, which can be done from the ECS task with any output you want. It's similar to activities, but the task token is pushed to downstream service instead of polling for it.

The output of a arn:aws:states:::ecs:runTask.sync task is the result of ECS DescribeTasks.

https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token

jogold
  • 6,667
  • 23
  • 41
adamwong
  • 1,035
  • 6
  • 11
1

I am currently working on a project that is leveraging Step Functions to launch multiple Fargate Tasks in sequence and I ran into this same issue. Currently there does not appear to be a way to return anything other than the standard ResponseSyntax although hopefully this is an area AWS will expand on.

Our work around for this limitation has simply been to use S3 to store the output from each Task so it can be accessed by subsequent Tasks (or other Steps). We provide the S3 bucket and S3 key prefix to each Step Function invocation as environment variables and since container overrides are returned as part of the ResponseSyntax they can be passed to additional Steps with the following snippet. 

"Overrides": {
    "ContainerOverrides": [
     {
       "Name": "<your container name>",
       "Environment.$": "$.Overrides.ContainerOverrides[0].Environment"
     }
    ]
}
Justin Newcom
  • 11
  • 1
  • Are you able to restart the pipeline from the point right after the exection of the Fargate Task? I assume you have to keep the S3 outputs forever. Make sure that they wont be overriden or you will loose the reproducibility. – Wertikal Jul 22 '19 at 14:53
  • We have not currently setup any restart support and our pipeline is only 2 steps for now. If you need to be able to run the pipeline from any point again and reproduce the results then yes you'd have to keep the S3 outputs around. Depending on how often you need to re-run jobs, S3's new intelligent tiering could help alleviate the costs of keeping files around in S3 for long periods of time. – Justin Newcom Jul 26 '19 at 16:28