14

I'm looking for better solution to handling our cron tasks in a load balanced environment.

Currently have:

  • PHP application running on 3 CentOS servers behind a load balancer.
  • Tasks that need to be run periodically but only on a single machine at a time.
  • Good old cron set up to run those tasks on the first server.
  • Problems if the first server is out of play for whatever reason.

Looking for:

  • Something more robust and de-centralized.
  • Load balancing the tasks so multiple tasks would run only once but on random/different servers to spread the load.
  • Preventing not having the tasks run when the first server goes down.
  • Being able to manage tasks and see aggregate reports ideally using a web interface.
  • Notifications if anything goes wrong.

The solution doesn't need to be implemented in PHP but it would be nice as it would allow us to easily tweak it if needed.

I have found two projects that look promissing. GNUBatch and Job Scheduler. Will most likely further test both but I wonder if someone has better solution for the above.

Thanks.

poisson
  • 1,324
  • 11
  • 20
  • The one-at-a-time can be solved by having a cron job on the most reliable server (or some other server entirely) execute the actual job on one of the servers at random via SSH. Since only one machine is responsible for the scheduling, you don't have to worry about sync/locking between all the servers. – Marc B Jun 23 '11 at 18:04

4 Answers4

6

You can use this small library that uses redis to create a temporary timed lock:

https://github.com/AlexDisler/MutexLock

The servers should be identical and have the same cron configuration. The server that will be first to create the lock will also execute the task. The other servers will see the lock and exit without executing anything.

For example, in the php file that executes the scheduled task:

MutexLock\Lock::init([
  'host'   => $redisHost,
  'port'   => $redisPort
]);

// check if a lock was already created,
// if it was, it means that another server is already executing this task
if (!MutexLock\Lock::set($lockKeyName, $lockTimeInSeconds)) {
  return;
}

// if no lock was created, execute the scheduled task
scheduledTaskThatRunsOnlyOnce();

To run the tasks in a de-centralized way and spread the load, take a look at: https://github.com/chrisboulton/php-resque It's a php port of the ruby version of resque and it stores the data in the same exact format so you can use https://github.com/resque/resque-web or http://resqueboard.kamisama.me/ to monitor the workers and see reports

Alex
  • 1,712
  • 19
  • 16
4

Assuming you have a database available not hosted on one of those 3 servers;

Write a "wrapper" script that goes in cron, and takes the program you're running as its argument. The very first thing it does is connect to the remote database, and check when the last time an entry was inserted into a table (created for this wrapper). If the last insertion time is greater than when it was supposed to run, then insert a new record into the table with the current time, and execute the wrapper's argument (your cron job).

Cron up the wrapper on each server, each set X minutes behind the other (server A runs at the top of the hour, server B runs at 5 minutes, C at 10 minutes, etc).

The first server will always execute the cron first, so the other two servers never will. If the first server goes down, the second server will see it hasn't ran, and will run it.

If you also record in the table which server it was that executed the job, you'll have a log of when/where the script was executed.

Corey Henderson
  • 7,239
  • 1
  • 39
  • 43
  • Thank you Corey. We already found some solutions working in a similar way. However this doesn't really meet our criteria and the only benefit compared to pure cron is the fallback on the 2nd or 3rd server if the first one goes down. The question wasn't so much about how to write that but rather whether there is already a tested mature solution out there. – poisson Jun 23 '11 at 22:25
  • I'm curious as to what you're using now? – Corey Henderson Jun 23 '11 at 22:26
  • We only use what I described in the Currently have section. That is cron trigerring required tasks directly on the first box. – poisson Jun 24 '11 at 09:08
0

Wouldn't this be an ideal situation for using a message / task queue?

jstats
  • 596
  • 4
  • 17
0

I ran into the same problem but came up with this litte repository: https://github.com/incapption/LoadBalancedCronTask

Chris
  • 1
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 16 '22 at 13:05
  • [Link only answers](https://meta.stackexchange.com/questions/8231/are-answers-that-just-contain-links-elsewhere-really-good-answers/8259#8259) are considered very low quality and [can get deleted](https://stackoverflow.com/help/deleted-answers), please put the important parts from the linked resource into the answer body. – helvete May 19 '22 at 15:43