0

I am working on a web application to provide a software as a web-based service using AWS, but I'm stuck on the implementation.

I will be using a Content Management System (probably Joomla) to manage user logins and front-end tasks such as receiving file uploads as the input. The program that provides the service needs to be run from the command line. However, I am not sure what the best way to automate this process (starting the program once the input file has been received) would be. It is an intensive program that will take at least an hour on each program, and should be run sequentially if there is more than one input at any one time, so there needs to be a queue where each element in the queue records the file path of the input file, the file path of the output folder, and ideally the email to send a notification to when the job is done.

I have looked into Amazon Data Pipeline and AWS Simple Workflow Service, and Simple Queue Services and Simple Notification Services, but I'm still not sure how exactly these could be used to trigger the start of the process, starting from the input file being uploaded.

Any help would be greatly appreciated!

sammtt
  • 401
  • 1
  • 6
  • 14

2 Answers2

0

There are a number of ways to architect this type of process; here is one approach what would work:

  1. On the upload, upload the file to an S3 bucket, so that it can be accessed by any instance later.

  2. Within the upload process, send a message to an SQS queue, which includes the bucket/key of the file uploaded, and the email of the user that uploaded it.

  3. Either with Lambda, or with a cron process on a purpose built instance(s), check the SQS queue, and process each request.

  4. Into the processing phase, add the email notification to the user when the process is complete.

Rodrigo Murillo
  • 13,080
  • 2
  • 29
  • 50
  • Thanks for your answer. Is it any different from this pattern: https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/ – sammtt Feb 02 '16 at 15:12
  • No - very similar to what I suggested. I like the ECS part. The only thing not considered there is the final email notification, which is trivial. – Rodrigo Murillo Feb 02 '16 at 15:32
0

You can absolutely use data pipeline to automate this process.

Take a look at managed preconditions and the following samples.

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-preconditions.html https://github.com/awslabs/data-pipeline-samples/tree/master/samples

Marc Beitchman
  • 1