15

My goal is to create an application that will be able to do long-lasting mainly system tasks, such as:

  • checking out code from the repositories,
  • copying directories between various localizations,
  • etc.

The problem is I need to prepare it somehow independently from the web browser. I mean that for example after starting the checkout/copy action, closing the web browser will not interrupt the action. So after going back to that site I can see that the copying goes on or another action started when the browser was closed...

I was searching through various tools, like RabbitMQ + Celery, Twisted, Pyro, XML-RPC but I don't know if any of these will be suitable for me. Has anyone encountered similar needs when creating Django app? Please let me know if there are any methods/packages that I should know. Code samples also will be more than welcome!

Thank you in advance for your suggestions!

(And sorry for my bad English. I'm working on it.)

Brian Neal
  • 31,821
  • 7
  • 55
  • 59
user1029968
  • 297
  • 2
  • 19

2 Answers2

11

Basically you need to have a process that runs outside of the request. The absolute simplest way to do this (on a Unix-like operating system, at least) is to fork():

if os.fork() == 0:
    do_long_thing()
    sys.exit(0)
… continue with request …

This has some downsides, though (ex, if the server crashes, the “long thing” will be lost)… Which is where, ex, Celery can come in handy. It will keep track of the jobs that need to be done, the results of jobs (success/failure/whatever) and make it easy to run the jobs on other machines.

Using Celery with a Redis backend (see Kombu's Redis transport) is very simple, so I would recommend looking there first.

David Wolever
  • 148,955
  • 89
  • 346
  • 502
  • 1
    Really thanks for your suggestions. I'll try Celery with Redis backend. Could you provide me a reliable sample or how-to with suggested way to implement it? I think I'll try these: http://query7.com/tutorial-celery-with-django http://ask.github.com/celery/tutorials/clickcounter.html Do they seem reliable? Any suggstions are more than welcome, thanks! – user1029968 Nov 04 '11 at 18:34
  • 1
    The second link is definitely reliable, as it's part of the celery documentation. In fact, I was able to get everything working with just the documentation, so I'd start with that. – David Wolever Nov 04 '11 at 18:49
  • Also, if our responses are helpful, you should hit the up arrow on the left of the answer to upvote them. – David Wolever Nov 04 '11 at 18:49
  • Sure, I know. Unfortunately these are my first posts at stackoverflow and I do not have enough reputation points to reward you, YET. – user1029968 Nov 04 '11 at 19:22
  • Ah, really? I thought it let you upvote answers to your own questions. On well. – David Wolever Nov 04 '11 at 19:32
  • The clickcounter tutorial may not be what you want. That is sort of an advanced tutorial using messaging together with Celery. You should check out the getting-started guides http://django-celery.readthedocs.org/en/latest/getting-started/first-steps-with-django.html (and if you want to use Redis see http://docs.celeryproject.org/en/latest/tutorials/otherqueues.html#redis) – asksol Nov 05 '11 at 21:10
6

You might need to have a process outside the request / response cycle. If that is the case, Celery with a Redis backend is what I would suggest looking into, as that integrates nicely with Django (as David Wolever suggested).

Another option is to create Django management commands, and then use cron to execute them at scheduled intervals.

Brian Neal
  • 31,821
  • 7
  • 55
  • 59
  • Thank you for your suggestion about management commands. I'll look through it if I'll fail with Celery + Redis solution will. Any other suggestions are welcome! – user1029968 Nov 04 '11 at 18:29