0

I am stucked in a case in which a single or multiplecompany can have multiple users and all users can work on a resource but only single user can utilize resource at once while other have to be wait. So I maintain user pending task data in table and check if any task state is pending by using celery function AsyncResult, if it is pending then do not spawn other task. Next, if task is successful then I query my table and fetch all pending tasks at once and put it in celery canvas function chain() to execute task in sequence.

I created a celery function in my Django API which execute for every request, and if a task is completed I deleted that task entry from table, so I wrote a django signal receiver function for post _delete which get more task for same company which the deleted task belongs.

Steps for the process I follow are:

  • Execute celery function
  • If there is no task execute the task for company, I created separate queue for every company dynamically.
  • If other task is there I only store them based on previous task state.
  • If a task is deleted then again using signal event I check if there is any other task for company for which task is deleted, If there is execute them in chain
  • So for every company only initial task get execute once from API, rest executed from signal event.

Now it worse, when I tested this scenario with 10 users, task get executed twice, sometimes executed simultaneously. Where I am doing wrong, please advise me.

Note: I chose celery because I have multiple companies scenario and their users can run their task in parallel. The resource contention only for multiple users in a single company for resources.

Shashank
  • 462
  • 1
  • 8
  • 26
  • can you think of any other ways to do this? like maybe a shared lock somewhere? what you're proposing above seems more complicated than it needs to be. tackle the first problem -- you have a shared resource that you only can allow one consumer to use. how to lock everyone else out that wants to use it? throwing Celery into the mix just complicates things. – matias elgart Nov 13 '16 at 15:22
  • Multiple company -> multiple users is the reason I go for celery, because for every company task should be run in parallel, but in sequence for users in company. I designed multiple queue for companies so the task can en-route in separate queue and utilise their own resources. Now sequence is required because for a resource at a time a single user can consume it, rest have to be wait. This is the case, but I don't think multiple queue is a that is the reason I post this scenario here. – Shashank Nov 14 '16 at 12:53
  • got it, that more sense. with this type of problem (you tried X, but Y seems to be happening), it's probably best if you include your celery task code so we can analyze in more detail. also include your relevant celery configuration. stay positive, you'll get it :) – matias elgart Nov 14 '16 at 13:31

0 Answers0