Is there a way I can queue long running tasks?

Question

Is there a way I can do the following in a unix terminal:

1. Start a long running process
2. Add another long running process to start when the previous is done
3. Repeat step 2 until I have queued the processes I need to have run

Reason I ask is that I have some long running stuff I need to do. I could put all the commands in a simple bash script and just execute that, but the problem is that I am not always sure exactly what I need to run. So if I start the script and then remember another one I should run, I need to hit ^C a bunch of times until all the processes are killed, edit the script and add my new process and then start the whole thing again.

What I specifically am doing right now is to copy a lot of large files on to various external hard drives, and since I don't know exactly which ones I need to copy and to where right now I'd like to start the ones I do know I need to copy, and then add to the queue as I figure out the rest.

Hope that made sense... Is anything like this possible?

Answer 1

The shell is perfect for that.

1. Start the first program.
2. Press Ctrl-z to suspend the program.
3. Start the next program.
4. Repeat steps 2 and 3 to add more programs to the queue. All the programs will be in Suspended mode.

5. Now run the queue like this:

   while fg; do :; done
   

   You may not suspend this while loop or exit bash until the queue is done.

If you will need to log out (e.g. the programs will run for many days) then consider running the above steps in screen.

Answer 2

Not very elegant, but quick and dirty:

process1 &
while kill -0 $! ; do sleep 1; done && process2 &
while kill -0 $! ; do sleep 1; done && process3 &

You would need to substitute actual PIDs for $! if you've run intervening background jobs.

Answer 3

The answer to your general question is: Absolutely. In the early days of computing, batch processing was the only way to do anything, and even when multi-user interactive systems were invented, batch-processing capability was the norm for large jobs. And it's still commonly done today in medium and large-scale environments using systems like Sun Grid Engine or Torque.

However, that's probably overkill for what you need. You could set up a more lightweight system to run scripts in a serial queue, but I don't think that approach is particularly well-suited to your specific task. Presuming parallel copies to different drives are acceptable, I think I'd attack it like this:

1. Create directory structure corresponding to your your target drives:

   ~/copysystem/drive1/ ~/copysystem/drive2/ ~/copysystem/drive3/

2. Install Incron.

3. Set up an incrontab entry for each of these directories, which runs your copy script automatically on IN_MOVED_TO.

4. Make your script either a) kill any previous instances of the same script when it starts or b) use a mkdir-based lockfile and block until the lock is cleared.

Then, all you need to do is move files to the various ~/copysystem/drive# directories, and they're all copied magically to your destination.

Especially in case of 4a, you probably want to use rsync -aP to copy your files, so that you can restart partial transfers from the middle. (Possibly in combination with --remove-sent-files, if you want to get rid of the originals.)

If you want to skip the complication of using incron, you can still take advantage of making your scripts block on a lock file. That works something like this:

#!/bin/bash
LOCKDIR="/var/run/copysystem/copysystem.lock"

while ! mkdir $LOCKDIR ; do
  echo "waiting for lock"
  sleep 5
done
trap rmdir $LOCKDIR EXIT

rsync commands go here....

This works because mkdir is an atomic operation — if it succeeds, you know the directory didn't exist. That's important, because if you use something like ! -f && touch, there's a race condition. (Same with scanning the process table for rsync commands, or the like.)

Answer 4

If this is something you'll be doing regularly, then its worth setting up some sort of scheduler to manage it for you.

I like the elegance of Aleksandr's solution - and it addresses all the points you originally raised - but I can see it does have limitations.

I've previously used the BSD lpd as a method for queueing jobs - the 'printer driver' is just a shell script so its easy to adapt to different tasks (in my case that meant managing 4 modems for polling data, sending faxes, SMS and other stuff).

Answer 5

What you want is a non-interactive command queue. Good news! I wrote one for you.

enqueue_cmd:

#!/usr/bin/perl -w

# Enqueue a job

use strict;

use Fcntl qw(:flock);

my $JOB_QUEUE_FILE = '/var/tmp/job_queue';
my $LOCK_TRIES = 5;
my $LOCK_SLEEP = 1;

my $jq;
open $jq, ">> $JOB_QUEUE_FILE" or die "!open $JOB_QUEUE_FILE: $!";

my $locked = 0;

LOCK_ATTEMPT: for (my $lock_tries = 0; $lock_tries < $LOCK_TRIES;
    $lock_tries++)
{
    if (flock $jq, LOCK_EX) {
            $locked = 1;
            last LOCK_ATTEMPT;
    }

    sleep $LOCK_SLEEP;
}

$locked or die "could not lock $JOB_QUEUE_FILE";

for (@ARGV) {
    print $jq "$_\n";
}

close $jq;

dequeue_cmd:

#!/usr/bin/perl -w

# Dequeue a jobs and run them

use strict;

use Fcntl qw(:seek :flock);
use FileHandle;

my $QUEUE_FILE = '/var/tmp/job_queue';
my $OUTPUT_FILE = '/var/tmp/job_out';
my $LOCK_TRIES = 5;
my $LOCK_SLEEP = 1;
my $JOB_SLEEP = 1;

my $locked;

my $jo;
open $jo, ">> $OUTPUT_FILE" or die "!open $OUTPUT_FILE: $!";
$jo->autoflush(1);

my $jq;
open $jq, "+< $QUEUE_FILE" or die "!open $QUEUE_FILE: $!";

my @jobs = ( );
my $job;

JOB: while (1) {

    if (-s $QUEUE_FILE == 0) {
            sleep $JOB_SLEEP;
            next JOB;
    }

    $locked = 0    

    LOCK_ATTEMPT: for (my $lock_tries = 0; $lock_tries < $LOCK_TRIES;
            $lock_tries++)
    {
            if (flock $jq, LOCK_EX) {
                    $locked = 1;
                    last LOCK_ATTEMPT;
            }

            sleep $LOCK_SLEEP;
    }
    $locked or die "could not lock $QUEUE_FILE";

    seek $jq, 0, SEEK_SET or die "could not seek to start of file";

    push @jobs, <$jq>;

    truncate $jq, 0 or die "could not truncate $QUEUE_FILE";
    seek $jq, 0, SEEK_SET or die "could not seek to start of $QUEUE_FILE";

    flock $jq, LOCK_UN;

    for $job (@jobs) {
            chomp $job;
            print $jo "## executing $job\n";
            print $jo `$job`;
    }

    sleep $JOB_SLEEP;
}

First, run nohup ./dequeue_cmd &, Then add your commands like so:

./enqueue_cmd "echo 'hello world'" "sleep 5" "date"
./enqueue_cmd "ls /var/tmp"

The output appears in /var/tmp/job_out:

tail -F /var/tmp/job_out
## executing echo 'hello world'
hello world
## executing sleep 5
## executing date
Fri Dec 10 16:35:43 PST 2010
## executing ls /var/tmp
ff
job_out
job_queue
ss