Imagine multi-level round-robin balancer: there are multiple teams, multiple users within a team, multiple sessions for a user, multiple jobs per session. We need some "fair" resource balancing between teams, users in a team, etc. In a classic approach, the job processor sends a token to highest level round-robin, which routes it to one of the lower non-empty round-robins, etc. until it reaches a particular job to do, whcih then goes to the job processor as a reply on a token, the next token is issued by the processor as soon as the job is done.
Now imagine that job processor is behind a network with quite a latency, so sending tokens one-by-one from processing side to aggregating side is not effective when jobs are quick to do, and sending tokens in bursts is not good when jobs are difficult.
The solution would be some dynamic reordering. We sequentally translate incoming job requests over the network, but tag them with some (minimalistic) information that would help reorder requests in a correct way on a job processor side.
In theory, we could send all the metadata for all job requests over the network and do all the round-robin scheduling on the processor side. But I'd want aggregating side to do most of reordering to minimise the network traffic. Besides, network bandwidth is limited too, so during waves of high load, it may be not possible to translate all incoming requests quick enough, so some partial ordering on aggregating side is essential.
