4

I would like to build an app, which would call periodic tasks at defined intervals - cron replacement like, but without spawning processes. I'll have a lot of tasks, around 10000 per minute.

  • I should be able to add tasks to scheduler dynamically, e.g using RPC
  • Tasks should be stored at some central place, like a database.
  • It should be scalable, meaning I could run something like worker instances that would process scheduled tasks - if one worker is already under heavy load, another one starts processing scheduled tasks.

I've looked into these options for now:

  • Celery - can work as a distributed system, but it's not really true scheduler. You can't schedule new jobs dynamically.

  • APScheduler - tasks can be added dynamically, can store tasks in the database but it's not distributed.

I can probabbly run seperate apscheduler instances, but then each would have different job store (mysql database or table). Also i'd need to determine manually, to which instance job should be submitted. Failover and scalability is also difficult.

Is there any existing python or non-python solution to this problem?

Josh
  • 81
  • 1
  • 3
  • 7

1 Answers1

2

Take a look at https://github.com/pditommaso/awesome-pipeline. This has a big list of workflow management tools, one of which is likely to meet your requirements.

For example: DoIt and JobLib, which declares themselves to be simple task system which parallel execution capabilities.

Many of them will be too heavy-weight for your needs, and some are more specialised (bioinformatics).

Examples of heavier task systems which run as management processes: Luigi, Toil, Airflow and Pinball.

The heavier ones will have a learning curve, but will allow you to manage tasks centrally.

Alan
  • 2,914
  • 2
  • 14
  • 26
  • Thanks, i will take a look. I can spend time learning, better than creating and managing own solution. – Josh Feb 28 '18 at 12:51
  • @Josh did you get any library that create jobs dynamically and distributed. I am also have same kind requirement. – Deepak Shinde Jul 30 '18 at 08:46