3

I have a single server shared among different users and I need to set up a simple workload manager/job scheduling system that would allow to share resources fairly among different users. What I'm looking for is a multi-user version of task spooler with some added functionality. Ideally, it should

  • allow different users to add job to one or multiple queues
  • each job should have a maximum walltime after which it is killed, and the next job in the queue is executed
  • handle "fairness" in the usage of resources (at minimum, it should cycle among different users)
  • allow for multiple job to run at the same time if need be (so if one user has a serial job the other users can take advantage of free resources)

I have found a number of job schedulers (as SLURM or Torque) that could fit the job, but they seem to have a steep learning curve and have a configuration and administration overheard which is way too big for a single server.

Any idea of what could work best?

tmt
  • 83
  • 4
  • Hi, this is an interesting question and questions in the same vein pop up on this site regularly. Yet it seems off-topic since it falls into the category of "Questions to recommend a tool". That means that relevance of answers would erode with time, which is not good. I suggest you formulate a question differently, e.g. "Is it possible to configure SLURM to manage a single host?" – Dima Chubarov Feb 20 '19 at 04:28
  • thanks, that's a good idea - but I'd rather not have to put up with SLURM anyway, it's a complicated piece of infrastructure which is not really designed for my needs. If people have a way to make it a bit more straightforward I'll be happy to try it however :) – tmt Feb 20 '19 at 07:54

1 Answers1

1

Celery might help. We use celery for running tasks both in single and distributed servers. We can modify as per our needs like adding UI with Flower

Irfan Junaid
  • 43
  • 8
  • Thank you - I'm having a look, it looks like it's not really designed to handle jobs as shell commands is it? – tmt Feb 20 '19 at 07:56