0

Ok I'm doing some testing with Stalker and Beanstalkd. My objective is to offload 500kb post requests to queue and process them asynchronously.

So far in my testing I have this very simple example.

#worker.rb
require 'stalker'
include Stalker

job 'hello' do |args|
  puts "hi"
  sleep 1
  puts "hello"
end

and this file for adding to queue

# stalker.rb
require 'rubygems'
require 'stalker'

10.times do 
    Stalker.enqueue('hello')
    puts 'queued'
end

So in one terminal I run

$ stalk worker.rb
Working 1 jobs: [ hello ]

Then I run the stalker file

$ ruby stalker.rb

stalker executes almost instantly as expected so no blocking on that.

but the worker takes approximately 10 seconds to run. Really I want this to be much closer to 1 second for those 10 jobs because I want them to run in parallel.

Any recommendations on the next best step to do this?

UPDATE: I've realised I can run multiple workers from different terminals and it will process the queue much faster e.g. 2 workers will do the process in roughly half the time.

Derek Organ
  • 8,323
  • 17
  • 56
  • 75

2 Answers2

1

As a heads up, you may want to check out the new gem I created called Backburner which is a more modern way to manage asynchronous jobs with beanstalkd in ruby. It follows a cleaner, resque-esque interface, has a truly asynchronous worker (that uses forking and multiple threads for much faster throughput) and has much better error handling and retry support. I was using stalker for 2 years before writing backburner because I wanted a better solution.

It gives you support for things like:

SomeObject.async(:pri => 1000, :delay => 2).some_method(1, 2, 3)

with the method then being automatically enqueued onto beanstalkd and processed asynchronously by backburner workers.

Nathan
  • 1,381
  • 20
  • 43
  • 58
0

Stalker.enqueue takes 3 options: job name string, arguments hash, options hash. in the option hash, you can pass the number of seconds that it takes to start . example:

Stalker.enqueue('hello',{},{:delay => 5})
Rodrigo Dias
  • 1,331
  • 2
  • 13
  • 18