Fork and Join with Amazon Lambda

Question

I am trying to write a lambda function that triggers many small lambdas

exports.handler = (event, context, callback) => {
   var AWS = require('aws-sdk');

   let noOfPages = 20, currentPage = 1;
   var message = '';
   //let payloadArray = [];
   while(currentPage <= noOfPages){
        message = '{"first_page": '+ currentPage +', "last_page": '+ (currentPage) + ', "file_name" : "something.doc"' +'}';
        console.log(message);
        var params = {
            FunctionName: 'test',
            InvocationType: 'Event',
            LogType: 'Tail',
            Payload: message
        };
        var convLambda = new AWS.Lambda({
            accessKeyId: 'key',
            secretAccessKey: 'secret',
            region: 'us-west-2'
        });

        convLambda.invoke(params, function(err, data) {
            if (err) {
                context.fail(err);
            } else {
                context.succeed('Processed : '+ data);
            }
        })
    currentPage+=1;        
   }
};

This works well and triggers, say 20 lambdas. I would however, like to wait till all the async lambdas are done (fork and join). Is there way to achieve this currently in Amazon Lambda?

found the answer I need under here http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/basics-async.html as java futures. — Jay, Jan 13 '17 at 10:22
The answer is of course very specific to to the programming language being used. The answer has very little to do with AWS Lambda, and is much more related to the available features of the specific programming language being used. As you found, you can use Java Futures if you are writing Lambda functions in Java. As hjpotter92 pointed out you would use Promises to accomplish the same thing in NodeJS. — Mark B, Jan 13 '17 at 17:17

score 6 · Answer 1 · answered Jan 13 '17 at 22:13

6

You could consider using AWS Step Functions, which can fork and join Lambda processes. It also shows the relationships in pretty graphs:

Step Functions

answered Jan 13 '17 at 22:13

John Rotenstein

241,921
22
380
470

2

If I understand AWS Step Functions correctly though you can't, for instance, trigger an arbitrary number of Lamda functions based on the size of the input and then block until they all finish. Do I have that right? – Nathan Bell Jan 24 '17 at 06:59

score 2 · Answer 2 · answered Jan 13 '17 at 18:04

You can use JavaScript Promises to achieve this using the AWS SDK for NodeJS:

exports.handler = (event, context, callback) => {
   var AWS = require('aws-sdk');

   let noOfPages = 20, currentPage = 1;
   var message = '';
   //let payloadArray = [];
   var convLambda = new AWS.Lambda({
       accessKeyId: 'key',
       secretAccessKey: 'secret',
       region: 'us-west-2'
   });

   let promises = [];
   while(currentPage <= noOfPages){
        message = '{"first_page": '+ currentPage +', "last_page": '+ (currentPage) + ', "file_name" : "something.doc"' +'}';
        console.log(message);
        var params = {
            FunctionName: 'test',
            InvocationType: 'Event',
            LogType: 'Tail',
            Payload: message
        };

        promises.push(convLambda.invoke(params).promise());
        currentPage+=1;        
   }
   Promise.all(promises).then((values)=>{
     callback(null, values);
   }).catch((err)=>{
     callback(err);
   });
};

Fork and Join with Amazon Lambda

2 Answers2

Linked