I have set of tasks and each of them requires some certain condition to be satisfied. For example task needs 100 Mb of RAM to be executed. Also I have multiple servers where workers are placed. Each worker has some configuration that describes how many resources it has. For example some worker is available to use 1 Gb of RAM for execution of task.
Is there any system for distribution tasks between workers so that each worker satisfies requirements of task passed to it? Or maybe I can somehow do it with message broker (RabbitMQ for example)?
I am aware writing my own application for this because I don't want to reinvent the wheel.
Workers are written in Go.
Asked
Active
Viewed 58 times
0
Nikita Lapkov
- 121
- 1
- 6
1 Answers
0
RabbitMQ doesn't explicitly know such things about it's clients. It knows how many clients are there, how many subscribers to the queue etc.
You could however set up the exchanges in the way that would distribute the tasks as desired. However, you would have to know before the publishing of the task to which (type of) subscriber should it go to.
cantSleepNow
- 9,691
- 5
- 31
- 42
-
but if some worker will fall it will cause loosing tasks – Nikita Lapkov Jun 22 '16 at 10:27
-
@NikitaLapkov it won't if you don't ACK until the job is actually done. – cantSleepNow Jun 22 '16 at 10:47
-
I mean if server "mike" will fall and I send task tagged as for server "mike" nobody will take it – Nikita Lapkov Jun 22 '16 at 10:49
-
Yes, you'd have to have multiple instances. RabbitMQ would keep the message until someone takes is (if you configure it that way). – cantSleepNow Jun 22 '16 at 12:05
-
and message will travel through all workers maybe even visiting some of them multiple times. I have thought about this scheme but isn't it too long in time? – Nikita Lapkov Jun 22 '16 at 12:11
-
I dont' understan what you mean with `message will travel through all workers maybe even visiting some of them multiple times.` Please update you question.. – cantSleepNow Jun 22 '16 at 12:12
-
I mean if I worker have rejected task, RabbitMQ will resend it to another random worker. This one can reject it too and this can be a loop – Nikita Lapkov Jun 22 '16 at 12:14
-
Yes, that's how it works. Please read up a bit on RMQ docs. There is also message TTL. – cantSleepNow Jun 22 '16 at 12:15