0

What i am trying to achieve Write a scheduler, that uses a database to schedule similar tasks at different timings.

For the same i am using celery beat, the code snippet below would give an idea

try:
    reader = MongoReader()
except:
    raise
try:
    tasks = reader.get_scheduled_tasks()
except:
    raise
celerybeat_schedule = dict()
for task in tasks:
    celerybeat_schedule[task["task_id"]] =dict()
    celerybeat_schedule[task["task_id"]]["task"] = task["task_name"]
    celerybeat_schedule[task["task_id"]]["args"] = (task,)
    celerybeat_schedule[task["task_id"]]["schedule"] = get_task_schedule(task)

app.conf.update(BROKER_URL=rabbit_mq_endpoint, CELERY_TASK_SERIALIZER='json', CELERY_ACCEPT_CONTENT=['json'], CELERYBEAT_SCHEDULE=celerybeat_schedule)

so these are three steps - reading all tasks from datastore - creating a dictionary, celery scheduler which is populated by all tasks having properties, task_name(method that would run), parameters(data to pass to the method), schedule(stores when to run) - updating this with celery configurations

Expected scenario given all entries run the same celery task name that just prints, have same schedule to be run every 5 min, having different parameters specifying what to print, lets say db has 

task name     , parameter , schedule
regular_print , Hi        , {"minutes" : 5}
regular_print , Hello        , {"minutes" : 5}
regular_print , Bye        , {"minutes" : 5}

I expect, these to be printing every 5 minutes to print all three

What happens Only one of Hi, Hello, Bye prints( possible randomly, surely not in sequence)

Please help, Thanks a lot in advance :)

mayank
  • 11
  • 3

1 Answers1

0

Was able to resolve this using version 4 of celery. Sample similar to what worked for me.. can also find in documentation by celery for version 4

    #taking address and user-pass from environment(you can mention direct values) 
    ex_host_queue = os.environ["EX_HOST_QUEUE"]
    ex_port_queue = os.environ["EX_PORT_QUEUE"]
    ex_user_queue = os.environ["EX_USERID_QUEUE"]
    ex_pass_queue = os.environ["EX_PASSWORD_QUEUE"]
    broker= "amqp://"+ex_user_queue+":"+ex_pass_queue+"@"+ex_host_queue+":"+ex_port_queue+"//"

    #celery initialization
    app = Celery(__name__,backend=broker, broker=broker)
    app.conf.task_default_queue = 'scheduler_queue'
    app.conf.update(
        task_serializer='json',
        accept_content=['json'],  # Ignore other content
        result_serializer='json'
    )
task = {"task_id":1,"a":10,"b":20}
##method to update scheduler
def add_scheduled_task(task):
    print("scheduling task")
    del task["_id"]
    print("adding task_id")
    name = task["task_name"]
    app.add_periodic_task(timedelta(minutes=1),add.s(task), name = task["task_id"])    

@app.task(name='scheduler_task')
def scheduler_task(data):
    print(str(data["a"]+data["b"]))
mayank
  • 11
  • 3