2

I have multiple producers that stage objects (jobs) for processing, and a single consumer that takes objects one-by-one. I need to design a sort of a scheduler in golang.

  1. Scheduling is asynchroneous, i.e. each producer works in a separate gorourine.
  2. Scheduler interface is "good" in terms of golang-way (I'm new in Go).
  3. A producer can remove or replace its staged object (if not yet consumed) with zero or minimal lost in the position in a queue. If a producer misses its slot because it canceled and then restaged an object, it still keeps a privilege to stage as soon as possible early till the end of the particular round.
  4. "Fair" scheduling between producers.
  5. Customizable multi-level weighting/prioritization

I'd like some hints and examples on right design of such a scheduler.

I feel I need every producer to wait for a token in a channel, then write (or don't write) an object to a shared consumer channel, then dispose the token, so it is routed to a next producer. Still, I'm not sure this is the best approach. Besides, it takes 3 sequential synchronous operations per producer, so I'm afraid I'll have performance pitfalls because of the token traveling too slowly between producers. Also, 3 steps for one operation is probably not a good golang-way.

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Roman Maltsev
  • 307
  • 1
  • 3
  • 9
  • I suggest you post some code about what you're trying to achieve. You can add comments to parts not clear to you but at least we have a direction to follow in order to not be OT. Thanks, man! – ossan Feb 14 '23 at 14:02

0 Answers0