1

Say I have a Lambda function called 'TestExecutor' which takes takes in an argument which contains ARNs for N 'Tests' which are also implemented as Lambda functions.

The workflow:

  1. TestExecutor is invoked with a list of ARNs of various 'Tests'
  2. TestExecutor calls each Test concurrently; each Lambda is expected to return a JSON
  3. TestExecutor waits for each Test to complete. It consolidates all the JSONs received
  4. Consolidated JSON is stored in DynamoDB/S3

Problem statement - What is the best way to create this kind of workflow in a Serverless manner?

I considered two AWS Services to manage this:

  1. AWS Step Functions - My step function would need states for each possible 'Test' Lambda that can be executed. I want to give flexibility to the user to invoke any Lambda without needing to 'register' it in my Step function.
  2. AWS SWF - Just seems a little overkill. Suffers from the same problem as above too.

So right now the best I can think of is doing this in a simple manner:

In my TestExecutor Lambda, I could create N threads for N tests each thread invokes a particular Test's Lambda function. Each thread waits for its Test to return a JSON. As all executions are successful, all JSONs are consolidated. Consolidated JSON is stored in DynamoDB.

I'm not happy with this solution - it will be a little tricky to manually manage failures and retries of the Test Lambdas from within the TestExecutor Lambda. This is my first time into trying something serverless, but it just seems like the wrong pattern. I'd like to get a nice top-down view of my workflow - it seems like monitoring this would be a little messy and scattered since there's no formal link between TestExecutor and the Test Lambdas

Maybe I could create an SQS Queue along with each Test Lambda. For each ARN supplied to the TestExecutor, I could push a message to a corresponding queue. But what now? I'd have to create 'Listener' Lambda's for each Test which polls each queue every T seconds. It would then invoke the actual Test Lambda. This also sounds needlessly complex.

Would love to hear some advice! Cheers.

user1265125
  • 2,608
  • 8
  • 42
  • 65

2 Answers2

1

AWS SWF doesn't suffer from the same problem as it doesn't require registration of a lambda function to invoke it. The main limitation of SWF is that it is still not possible to run decider process as a lambda function. So you'll have to run it somewhere else. If you already have some host that can run it implementing your use case using AWS Flow Framework is pretty straightforward.

Maxim Fateev
  • 6,458
  • 3
  • 20
  • 35
  • Of course I can invoke a Lambda function without 'registration'. But then what is the point of using SWF? What is the value addition? I'll still have 1 'TestExecuter' Lambda running 'invisible' Test Lambdas in the background. Won't I still suffer from the same issues as in the case of Step Functions? I'll read up some more on SWF. Also, I have no intention of using Flow framework since there's no Python SDK. – user1265125 Mar 11 '17 at 07:09
  • The value add is that decider function is stateless. So any failures of the process that host it are not going to lead to loss of the progress state. SWF is also fully transactional and doesn't produce duplicates (like SQS). So writing the decision logic is much simpler. If you were using Java then your use case would be coded in a dozen lines of code. – Maxim Fateev Mar 13 '17 at 00:51
  • What if I have an SQS queue, and my own 'decider' TestExector lambda which polls the queue every 5 seconds, and runs whatever Tests are required? It seems to me this will be even shorter and simple. Could you elaborate on the 'SWF is also fully transactional and doesn't produce duplicates' part? – user1265125 Mar 14 '17 at 16:34
  • AFAIK the 'duplicates' problem with SQS is that we cannot avoid duplicate messages from being inserted into SQS. That is a problem I can take care of at the client layer which populates SQS. – user1265125 Mar 14 '17 at 16:35
0

You could leverage the AWS SDK to generate a Step Machine using said ARNs from within a Lambda Function.

It would require some way to clean up afterwards somehow, and / or avoid duplicates, or the console would quickly get messy.

ElFitz
  • 908
  • 1
  • 8
  • 26