0

I am creating an AWS step function where one of the step, let's call it step X, starts a variable number of lambdas. Since these lambda functions are long (they take between 1 and 10 minutes each to complete), I don't want to wait for them in step X. I would be spending money just for waiting. I therefore start them with InvocationType=Event so that they all run asynchronously and in parallel.

Once step X is done starting all these lambdas, I want my step function to wait for all these asynchronous functions to complete. So, a little like described here, I would create some kind of while loop in my step function. This loop would wait until all my asynchronous invocations have completed.

So the problem is: is it possible to query for the status of an AWS lambda that was started with InvocationType=Event?

If it is not possible, I would need my lambdas to persist their status somewhere so that I can poll this status. I would like to avoid this strategy since it does not cover problems that occur outside of my lambda (ex: out of memory, throttling exceptions, etc.)

mabead
  • 2,171
  • 2
  • 27
  • 42

3 Answers3

1

An asynchronously invoked Lambda is a "fire and forget" use case. There's no straightforward way to get its result. I'm afraid you'll have to write your own job synchronization logic.

Milan Cermak
  • 7,476
  • 3
  • 44
  • 59
0

instead of polling,(which again is expensive), you can provide a callback, for the lambda to post back asynchronously. once you get all positives for all lambdas, then continue the process.

Jackie
  • 25,199
  • 6
  • 33
  • 24
  • How to do that? My use case is simple, start a lambda, wait, if its a success then exit, if its a failure then fire a message to a SNS topic. – Aakash Basu May 04 '20 at 10:40
  • You can now do that using lambda destinations : https://aws.amazon.com/en/blogs/compute/introducing-aws-lambda-destinations/ – mabead May 04 '20 at 11:24
0

Since the question was initially posted, AWS added the support for dynamic parallelism in workflows. The need to manually start lambda functions and poll for their completion from within a step function is therefore now an anti-pattern.

mabead
  • 2,171
  • 2
  • 27
  • 42
  • My use case is simple, start a lambda, wait, if its a success then exit, if its a failure then fire a message to a SNS topic. – Aakash Basu May 04 '20 at 10:40