1

We're trying to move to AWS and to use DynamoDB. It'd be nice to keep everything under DynamoDB so there aren't extraneous types of databases, but aside from half complete research projects I'm not really finding anything to use for a scheduler. There's going to be dynamically set schedules in the range of thousands+, possibly with many running at the same time. For languages, Java or at least JVM would be awesome.

Does anyone know a good Scheduler for DynamoDB or other AWS technology?

---Addendum When I say scheduler I'm thinking of something all purpose like quartz. I want to set a cron and it runs at that time with the code I give it. This isn't doing some AWS task, this is a task internal to our product. SWF's cron runs inside the VM, so I'm worried what happens when the VM is down. Data Pipeline seems a bit too much. I've been looking into making a dynamodb job store for quartz, consistent read might get around the transaction and consistency issues, but I'm hesitant, might be biting off a lot with a lot of hard to notice problems.

nilacqua
  • 167
  • 3
  • 15

2 Answers2

3

Have you looked at AWS Simple Workflow? You would use the AWS Flow Framework to program against the service, and they have a well documented Java API with lots of samples. They support continuous workflows with timers which you can use to run periodic code (see code example here). I'm using SWF and the Flow Framework for Ruby to run async code that gets kicked off from my main app, and it's been working great.

Another new option for you is to look at AWS Lambda. You can attach your Lambda function code directly to a DynamoDB table update event, and Lambda will spin up and shut down the compute resources for you, without you having to manage a server to run your code. Also, recently, AWS launched the ability to call the Lambda function directly -- e.g. you could have an external timer or other code that triggers the function on a specific schedule.

Lastly, this SO thread may have other options for you to consider.

Community
  • 1
  • 1
readyornot
  • 2,783
  • 2
  • 19
  • 31
  • 1
    I would also add AWS Data Pipeline, for example for import and export of data in and out of DynamoDB: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb.html – Guy Apr 18 '15 at 07:49
  • I coded a version that runs using simple workflow. I'm concerned about a few things that is leading me to want a change. (1) The simpleworkflow cron job(not periodic timer) is a cron job within the VM not Amazon Land. It could get lost if the VM goes down and isn't back up in time for the scheduled to finish timeout. (2) Because it's a cron job in the VM, I'm worried about what happens when there's a large number of schedules. Will maintenance be a disaster. Will the VM get slowed down by so many crons. We're talking a very large number of jobs. I'm putting more info in the original post. – nilacqua Apr 20 '15 at 19:49
  • I'm sorry, but I don't know much about running cron jobs in AWS... I've had mixed results getting replies on the AWS Developer Forums, but it might be worth posting your question there: [EC2 Forums](https://forums.aws.amazon.com/forum.jspa?forumID=30&start=0) & [SWF Forums](https://forums.aws.amazon.com/forum.jspa?forumID=133). – readyornot Apr 21 '15 at 22:00
2

Another option is to use AWS Lambda Scheduled Functions (newly announced on October 8th 2015 at AWS re:Invent).

Here is a relevant snippet from the blog (source):

Scheduled Functions (Cron)

You can now invoke a Lambda function on a regular, scheduled basis. You can specify a fixed rate (number of minutes, hours, or days between invocations) or you can specify a Cron-like expression:

AWS Lambda scheduled functions

mkobit
  • 43,979
  • 12
  • 156
  • 150
  • 2
    I was hoping for something programmatic, but this does answer the question. We ended up using Quartz. – nilacqua Oct 09 '15 at 16:46
  • 1
    With Quartz, you still have a single point of failure, correct? What did you do to avoid SWF's problem mentioned in your question (VM going down)? – atlantis Mar 15 '16 at 10:50