5

hi
i'm going to set up a rails-website where, after some initial user input, some heavy calculations are done (via c-extension to ruby, will use multithreading). as these calculations are going to consume almost all cpu-time (memory too), there should never be more than one calculation running at a time. also i can't use (asynchronous) background jobs (like with delayed job) as rails has to show the results of that calculation and the site should work without javascript.
so i suppose i need a separate process where all rails instances have to queue their calculation requests und wait for the answer (maybe an error message if the queue is full), kind of a synchronous job manager.

does anyone know if there is a gem/plugin with such functionality? (nanite seemed pretty cool to me, but seems to be only asynchronous, so the rails instances would not know when the calculation is finished. is that correct?)
another idea is to write my own using distributed ruby (drb), but why invent the wheel again if it already exists?

any help would be appreciated!

EDIT: because of the tips of zaius i think i will be able to do this asynchronously, so i'm going to try resque.

user573335
  • 350
  • 1
  • 5
  • 18

2 Answers2

5

Ruby has mutexes / semaphores.

You can use a semaphore to make sure only one resource intensive process is happening at the same time.

However, the idea of blocking a front end process while other tasks finish doesn't seem right to me. If I was doing this, I would use a background worker, and then use a page (or an iframe) with the refresh meta tag to continuously check on the progress.

That way, you can use the same code for both javascript enabled and disabled clients. And your web app threads aren't blocking.

zaius
  • 6,409
  • 4
  • 28
  • 50
  • that metarefresh thing would be possible, but having the user to manually reload the page if the computation is not finished after the refresh isn't too userfriendly in my eyes. atm i'm not expecting too many requests at the same time, so blocking the web threads isn't a big concern. but you're right, thats not a nice way to do it. – user573335 Jan 13 '11 at 14:26
  • You can do the meta refresh as many times as you want - if the job isn't finished after the first refresh, show them the same refreshing page, and only show the final page once the job is done. – zaius Jan 13 '11 at 21:15
  • well, i'm a bit hesitant about it, but the more i think about it the more it seems the better solution to do it asynchronously, at least in the long run. are there job queues which i can ask about the expected task duration (from a web thread) and which can return an error message to the web thread if there are too many jobs queued? – user573335 Jan 14 '11 at 12:03
  • Expected task duration depends completely on what you're trying to do, and that's a whole question of its own. But you could update the status of a job with any type of interprocess communication. And yes, most queues will let you check the queue length before you enqueue a job. I personally use resque. – zaius Jan 15 '11 at 01:19
  • resque looks very interesting, but like most of these job queues, there seems to be surprisingly little documentation (or i simply can't find it). do you have any hints on where to look? in particular, i want to do the following: if i enqueue a job, the method should return if succesful. to do that, i need to implement a method for the queue to estimate total duration of all queued tasks (i know how to calculate that from job details); then the queue should deny new jobs if its already too busy. any idea if and how that is possible? – user573335 Jan 15 '11 at 16:54
  • Everything you need to know about resque you can find in the repo readme: https://github.com/defunkt/resque You can make a wrapper for adding to the queues that checks the queue length before it allows you to add to it. – zaius Jan 16 '11 at 03:49
  • My pleasure. It's an interesting question. Update me with what you end up doing. And good luck! – zaius Jan 16 '11 at 21:00
1

If you have a separate process, then you have a background job... so either you can have it or you can't...

What I have done is have the website write the request params to a database. Then a separate process looks for pending requests in the database - using the daemons gem. It does the work and writes the results back to the database.

The website then polls the database until the results are ready and then displays them.

Although I use javascript to make it do the polling.

If you really cant use javascript, then it seems you need to either do the work in the web request thread or make that thread wait for the background thread to finish.

To make the web request thread wait, just do a loop in it, checking the database until the reply is saved back into it. Once its there, you can then complete the thread.

HTH, chris

Chris Kimpton
  • 5,546
  • 6
  • 45
  • 72
  • ok, do you know how i can make the web request thread wait for the background job to finish? since the web reuest thread can't spawn the background thread, i don't know how to do that. – user573335 Jan 13 '11 at 08:39