nsq go client can't keep up

Question

I'm a Go novice coming from the Node world and I'm building a consumer using the official Bitly Go client. I am using AddConcurrentHandlers to spawn 50 goroutines to handle the fire hose of messages. The issue is that my consumer falls behind leaving an exponential of unprocessed/received messages on nsq. Has anyone else encountered this?

I built the same thing in Node to see if there was a server or NSQ configuration issue and its able to process all messages as quickly as they come in.

GO CODE:

q, _ := nsq.NewConsumer("chat", "golangbetches", config)

q.AddConcurrentHandlers(nsq.HandlerFunc(func(message *nsq.Message) error {
  l.Debug("Got a message: %v", message)
  message.Finish()
  return nil
}), 50)

err := q.ConnectToNSQLookupd("<address here>")

You mean the int I give as the second param in AddConcurrentHandlers? Its required by the api, probably to prevent an infinite number of routines — Walter Cecil Worsley IV, Mar 24 '15 at 21:44
No, JimB asked about your environment variable GOMAXPROCS. Unset or too low won't use all cores. Try setting it to 2, 4, 8 and 12 and observe your application. — Volker, Mar 24 '15 at 21:50
@cworsley4: GOMAXPROCS sets the maximum number of active threads running goroutines in your application. — JimB, Mar 24 '15 at 21:52
Yeah I'm a real noob. I hadn't set that env var so it maxes out at 6 right? I upped it to 70 and I'm seeing the same issue. — Walter Cecil Worsley IV, Mar 24 '15 at 21:58
Well, setting `n` to some random high number just isn't the best solution with most things `n`. More info on GOMAXPROCS at the summary of http://golang.org/pkg/runtime/ and http://golang.org/pkg/runtime/#GOMAXPROCS . It looks like you're using the [go-nsq](https://github.com/bitly/go-nsq) package; in which case, you might not be using it correctly per the [examples I just looked at](https://github.com/bitly/go-nsq/blob/master/UPGRADING.md#improving-the-nsqhandler-interface) — Momer, Mar 24 '15 at 22:20
I believe you are correct @Momer. So I added inflight to my NSQ config and set to 1000 (from the example). That fixed the issue. Do you have any insight into why that makes a difference? — Walter Cecil Worsley IV, Mar 24 '15 at 23:21
I'm not familiar with the project, but it seems that MaxInFlight would be the max number of messages in flight at any given time over all consumers. It defaults to 1, so that could be your resolution: https://github.com/bitly/go-nsq/blob/edb565821d2ffe9b8927d2c4a6cc8437c476ccc8/config.go#L172-L173 — Momer, Mar 24 '15 at 23:52

score 4 · Accepted Answer · answered Mar 24 '15 at 23:57

4

cfg.MaxInFlight handles the, "maximum number of messages this comsumer instance will allow in-flight..." More details are available in the consumer source

Set cfg.MaxInFlight to something reasonable, as it defaults to 1

An example configuration is available in the documentation where it is set to 1000. This may or may not be suitable for your application; and, you'd do well to monitor it, as a misconfiguration may result in truncated messages.

answered Mar 24 '15 at 23:57

Momer

3,158
22
23

There should absolutely be more clear documentation around this. – Walter Cecil Worsley IV Mar 25 '15 at 00:02
Make a PR, I'm sure they'll accept some enhancements to the docs! – Momer Mar 25 '15 at 00:07
Interesting point; I created the same server in Node without specifying a MaxInFlight option and it was able to keep up. And the Node module I used (nsq.js) defaults to 1. Strange – Walter Cecil Worsley IV Mar 25 '15 at 00:10
Well, interestingly enough, the `reader.js` in the js module for nsq has a default of `10` which overrides the `connection.js` of `1`. More info @ https://github.com/segmentio/nsq.js/blob/badb32bb2310084decec05f1cf85c49c70517ec9/lib/reader.js#L60 and https://github.com/segmentio/nsq.js/blob/badb32bb2310084decec05f1cf85c49c70517ec9/lib/connection.js#L50 – Momer Mar 25 '15 at 00:19
I'm having trouble understanding why MaxRdyCount < MaxInFlight could lead to truncation. Isn't it fine for it to be less? I'd think > would be the worse problem. – tunesmith May 12 '17 at 00:29

score 0 · Answer 2 · answered Nov 22 '16 at 08:06

Another ways to speed up go nsq consumers can be found here: https://github.com/nsqio/go-nsq/issues/187

One of them, which works for me, is increasing --max-rdy-count on nsqd, which allows to increase consumer's --max_in_flight even higher than the default 2500.

nsq go client can't keep up

2 Answers2