0

I'm trying to use Backburner (a Ruby client for Beanstalkd) in my application to create, say, 50 workers all watching the same tube for incoming jobs. The problem is, each of those 50 workers, when performing a job, does some heavy file manipulation, etc., and each of them needs to perform some operations on a uniquely-named numbered directory. For example, worker 7 needs to work with a directory named dir7.

Initially I thought I could make a ThreadsOnFork worker with 50 threads, and I'd be able to access a number associated with each thread from the worker's perform() method, but I haven't been able to do this yet.

How do I handle this situation using Backburner? Or, if it's not possible with Backburner, using some other Ruby beanstalkd client? I hope I'm getting my question across clearly enough. Thanks.

Vicky Chijwani
  • 10,191
  • 6
  • 56
  • 79
  • If the worker just needs a unique place to work on files, why not just create a new directory when you start up the worker process, based on (for example) a GUID, and then delete it when the worker exits. – Alister Bulman Feb 19 '13 at 11:28
  • @AlisterBulman I solved the issue some other way. I've added my solution as an answer below. – Vicky Chijwani Feb 19 '13 at 16:32

1 Answers1

0

I solved the issue, the crude way: wrote a shell script that starts a worker by passing its designated number to it via environment variables. Below is a snippet, if anyone's looking for a similar solution:

#!/bin/bash
for i in $(seq 1 "$1"); do
    env WORKER_NUM=$i rake QUEUES=queue1 backburner:work > logs/backburner.log 2>&1 &
done

The rake task becomes available by adding require 'backburner/tasks' to the Rakefile, as explained in the Backburner documentation.

Vicky Chijwani
  • 10,191
  • 6
  • 56
  • 79