Execute jobs concurrently in sequential order

Question

"What?" you ask, "That title doesn't make any sense."

Consider the following: Jobs with different ids may be processed asynchronously but jobs with the same id should be processed synchronously and in order from the queue.

My current implementation creates a go routine to handle the jobs for each specific id and looks something like this:

func FanOut() chan<- *Job {
    channel := make(chan *Job)
    routines = make(map[string]chan<- *Job)
    go func() {
        for j := range channel {
            r, found := routines[j.id]
            if !found {
                r = Routine()
                routines[j.id] = r
            }
            r <- j
        }
    }()
    return channel
}

This appears to work well (in current testing) but the creation of thousands of go routines might not be the best approach? Additionally the fan out code blocks unless a buffered channel is used.

Rather than a collection of go routines (above) I'm considering using a collection of sync.Mutex. The idea would be to have a pool of go routines which must first establish a lock on the mutex corresponding to the job id.

Are there any existing Go patterns suited to handling these requirements?

Is there a better approach?

There isn't really a "too many goroutines". Internally, Google runs services that spawn millions of goroutines. They are designed to be lightweight. — Adrian, Apr 17 '19 at 15:06
You might find this informative: https://stackoverflow.com/questions/8509152/max-number-of-goroutines — Adrian, Apr 17 '19 at 15:11
Given the lightweight nature of go routines would you say that the design here is reasonable then? — rtev, Apr 17 '19 at 15:13
I don't have enough information to say that but based on what's presented I wouldn't immediately call it unreasonable. I would assume there's no problem unless you measure and detect a problem. — Adrian, Apr 17 '19 at 15:15
Is the cardinality of your `id` bounded/fixed? Or can it grow dynamically? Thousands of go routines shouldn't be an issue, but having an unbounded operation could cause issue (ie `if !found`) — dm03514, Apr 17 '19 at 17:40
It's bound a practical limit of several thousand. The current implementation works well, but it got me thinking about an implementation that could handle an unbounded situation. — rtev, Apr 17 '19 at 19:16

AJR · Answer 1 · 2019-04-30T23:25:03.777

Create a channel for each ID - perhaps a slice of channels or a map (indexed by ID). Each channel would have a go-routine that processes the jobs for that ID in order. Simple.

I wouldn't worry about creating too many go-routines. And I wouldn't use mutex - without getting into too much detail using channels and go-routines allows each job to only be processed by one go-routine at a time and avoids possibility of data races.

BTW I only added this as an answer as I am not permitted to add comments (yet?).

Execute jobs concurrently in sequential order

1 Answers1