Scheduling jobs from a web environment on Linux

Question

I am developing an application in PHP on Linux/Apache. I want to be able to schedule PHP jobs (scripts) for execution at some specific time in the future from within the application.

I know that many people will recommend cron and at, but first of all I don't need recurrence (cron) and secondly and most importantly, I need the solution to be able to scale. At was not designed with race condititions in mind, and if two users try to add a job at the same time one or both may fail.

It's also important that jobs are executed at their specified time, and not just 'polled' once per minute or so.

Can anyone please suggest solutions for this task? Thank you.

Why not use cron/at and take care of the concurrency issues inside the script? — Pekka, Dec 25 '10 at 12:09
I think running a lot cron/at(s) could have scaling issues because there are going to be forked a lot processes. — Alfred, Dec 25 '10 at 14:41

score 2 · Answer 1 · edited Dec 25 '10 at 15:08

write a php-script as a daemon the script runs in the background, wakes up every some microseconds, checks for tasks to execute. if it has some tasks to execute, it could then fork itself and start the task.

if you don't like, that the script wakes up every some microseconds, you could also calculate the usleep time to the next task and then fork to start it. when tasks are being added or deleted, just send signal to the program and it then wakes up and recalculates how long it has to sleep now.

hyperlinks:

http://en.wikipedia.org/wiki/Daemon_%28computer_software%29

http://php.net/manual/en/function.pcntl-fork.php

http://www.php.net/manual/en/function.pcntl-signal.php

I upvoted you because of PHP-only solution. But I think if you can (re)compile those modules because standard the aren't enabled I would go for better scaling alternative like for example beanstalkd. — Alfred, Dec 25 '10 at 15:33

score 0 · Answer 2 · answered May 28 '14 at 17:34

On POSIX systems, there should be the at-daemon which is basically a one shot version of cron:

$ at 13:20 executeVeryImportantScript.php withPositionalParameters taken

which could be called via php's exec (or whatever is used to execute shell commands via php).

Reimplementing already existing solutions is Very Bad Practise ™:

You need to implement it, which is not cost efficient
You need to test the code
You need to maintain the code

score 0 · Answer 3 · answered Jul 13 '15 at 21:20

Here is a solution for Yii2 using MySQL. If you put the query/status update in a transaction, you can have multiple workers on different machines.

cron table

Field         | Type        
--------------+-------------
id            | int(11)     
execute_after | timestamp   
executed_at   | timestamp   
status        | int(11)     
progress      | int(11)     
action        | varchar(255)
parameter     | varchar(255)
result        | varchar(255)

Cron class

  /** This command waits for jobs to work on by polling the DB every 10s. TODO put this into Redis.
   * 
   */
  public function actionIndex()
  {
    do {
      $job = Cron::find()->where('execute_after < :now AND status = 0 ORDER BY id ASC', [':now'=>date(DATE_ATOM)])->one();
      if($job && method_exists($this, $job->action)){
        $job->status = 1;
        $job->save() or Yii::error($job->getErrors());
        $result = $this->{$job->action}($job, $job->parameter);
        $job->status = 2;
        $job->progress = 100;
        $job->executed_at = date(DATE_ATOM);
        $job->save() or Yii::error($job->getErrors());
      }
      sleep(10);
    } while (true);
  }

Then you can have tasks in the class like

function toggleStatus($job, $id) {

which do the work and update the progress of the job as it completes.

score 0 · Answer 4 · answered Dec 25 '10 at 13:42

0

you can try gearman

answered Dec 25 '10 at 13:42

Ivan

2,262
1
18
16

I don't believe can schedule? – Alfred Dec 25 '10 at 14:41

score 0 · Answer 5 · answered Dec 25 '10 at 14:29

0

This is what you could(I would) do:

Use Google's app engine Task Queue. It has a generous free quota(also pricing plane aren't that bad), it scales, uses webhooks. I think this is the easiest solution to implement.
beanstalkd.

answered Dec 25 '10 at 14:29

Alfred

60,935
33
147
186

I am curious how I would use a queue like beanstalkd for this? As far as I can tell, beanstalkd does not let me specify a time to execute the job - it's just a FIFO message queue? – Anders Feder Dec 27 '10 at 07:12
You should read the protocol. https://github.com/kr/beanstalkd/raw/v1.3/doc/protocol.txt. It says you can do a "put with delay". – Alfred Dec 27 '10 at 13:37
You should have enough workers to handle the load else it won't get executed in time. – Alfred Dec 27 '10 at 17:40

Scheduling jobs from a web environment on Linux

5 Answers5