I am trying to develop a micro-service system capable of managing queues of tasks in a parallel and highly efficient way, but I am stuck.
I briefly describe how the system should behave.
The system is made up of n micro-services, each of which manages a specific task. Each micro-service can pass a list of tasks to any of the other micro-services.
To allow a large number of tasks managed by a single micro-service, I thought of creating a master micro-service that takes as input (for example from a sort of bus) the tasks to be executed, and gives them to be processed by n ° its workers, splitting them into pieces. The split is performed in elements of different sizes, eg. 3 workers: 1st task manages 100 items, 2nd task manages 2 items, 3rd task manages 20 items. When a worker finishes its work, the master micro-service makes the result available and returns the result to the requesting micro-service (or to the bus), checks if there are other tasks to be performed and if there are, it passes them to the first worker that he freed himself.
I tried Nameko and Celery but I can't develop it in a way that meets these needs, especially for the asynchronous management part of the work managed by the worker which returns the result to the master which instantly makes the result available to the bus (or to the requestor) and passes the free worker another list of tasks to perform
