0

I want to use django celery as a bot of sorts that continuously loops through objects and gets data for them.

For example, lets say I have a huge database of movies. I want to loop through all the movies, one by one, and get their metadata. If a user said they wanted a particular movie metadata refreshed, I want to push this movie from whatever position on the queue to the front, then bring it back to the end of the queue once it has finished getting the metadata.

A lot of sites showed me how to schedule tasks that ran on specific time intervals, but I need the jobs to go one after another, and the metadata download time is going to vary between movies (I have a timeout if it's taking too long).

Does anybody know how I can implement something like this? Thank you!

takeshibaconsuzuki
  • 67
  • 1
  • 8
  • 1
    Just wrap your celery task in an infinite loop. – Jand Nov 16 '15 at 05:54
  • @Jand how would I implement the job queue so that any view can push to it? – takeshibaconsuzuki Nov 22 '15 at 20:19
  • @Jand But in this way, you cannot exceed the number of your OS threads for your celery tasks. For example, I create four endless celery tasks and my machine has four threads, then the fifth task isn't created and remains in the pending (`received`) phase. – Benyamin Jafari Jan 04 '22 at 08:24

0 Answers0