Stack Overflow
Stack Overflow

Questions tagged [nsq]

NSQ is a realtime message processing system designed to operate at bitly’s scale, handling billions of messages per day.

NSQ is a realtime message processing system designed to operate at bitly’s scale, handling billions of messages per day.

It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Go and Python libraries are available out of the box.

Questions can also be posted to the nsq-users Google group.

Bugs can be filed on GitHub.

59 questions
2
votes
1 answer

Should all independent tasks be processed outside the http request goroutine?

Since golang handles incoming requests in separate goroutines, it's unclear to me which types of tasks should be deferred for processing by a message queue e.g. NSQ consumers and which should be handled within the http request goroutine.
paulkon
  • 1,755
  • 2
  • 20
  • 34
2
votes
2 answers

nsq go client can't keep up

I'm a Go novice coming from the Node world and I'm building a consumer using the official Bitly Go client. I am using AddConcurrentHandlers to spawn 50 goroutines to handle the fire hose of messages. The issue is that my consumer falls behind…
Walter Cecil Worsley IV
  • 464
  • 1
  • 11
  • 20
1
vote
1 answer

Graceful Downscale GCE Instances

I have a usecase in which i'm using GCE instances as NSQ consumers to work on async processes. This hostgroup have an autoscaling configured when the CPU is above 60% which usually gets triggered when there were alot of messages being consumed at…
Radhian Amri
  • 327
  • 1
  • 3
  • 11
1
vote
1 answer

Communicate with NSQ over a Websocket

Is it possible to communicate with NSQ over a websocket connection from a web app running in a user's browser? Any built-in NSQ features or third-party library suggestions would be of a great help.
lig
  • 3,567
  • 1
  • 24
  • 36
1
vote
1 answer

Inject dynamically commands into a deployment

When you have a StatefulSet or a Deployment you usually have means to provide commands that will be fed into a pod's container on its creation. Is it possible, more arguments to be injected into the commands array as the deployment autoscales with…
KDX2
  • 945
  • 2
  • 14
  • 31
1
vote
1 answer

How to get docker generated ip port from docker-compose

docker networking seems not working, docker supposed to know the address and port when specifying the container name, or what am I missing? Would it be possible to get the generated ip port like: docker-compose.yml command: nsqd …
swoopy
  • 326
  • 4
  • 18
1
vote
0 answers

How to connect/listen to nsqd from a webserver

So I have a simple flask server and wanted to listen to nsqd,(My nsqd and nsqlookupd dockers are running fine.) I'm using gnsq from python as a tool to achieve that server.py: from flask import Flask import gnsq consumer =…
swoopy
  • 326
  • 4
  • 18
1
vote
1 answer

Which --broadcast-address should i set in nsqd param?

Now i am starting nsqd with nsqd --max-msg-timeout=1800000 --lookupd-tcp-address=127.0.0.1:4160 I am using nsqjd for nodejs with nsq. There was everyhing ok until some days. Now, when starting nodejs with nsqjs in it i receive error: Error:…
Tyler Brin
  • 59
  • 1
  • 4
1
vote
1 answer

Is the TimerThread cause DeadLock happend?

Timer-2 thread is waiting on the 0x00000000e1a23398, and it's also locking the 0x00000000e1a23398. Will this situation cause Timer-2 thread deadlock? Can anybody explain it for me?
Kay
  • 59
  • 7
1
vote
2 answers

NSQ Re-Queue half Message

I have an array of Ids of type int64 And this is my Nsq Message that I am trying to publish. nsqMsg := st{ Action : "insert", Ids : Ids GID : Gids } msg, err := json.Marshal(nsqMsg) if err != nil { …
Kartik Khandelwal
  • 11
  • 1
1
vote
0 answers

Why were the messages delayed in NSQ? It's unexpected……

I used the NSQ as the mq in my project，java producer produce the message to NSQ and go consumer consume it.But the strange things is that the consumer always get the message after few seconds.There is just a few messages,I really don't know how to…
Siyong Xiao
  • 21
  • 3
1
vote
0 answers

Move Data from Elastic Search to NSQ and then to a Mysql database

I have a use case where we are using snowplow analytics to get data from various api's and webpages , due to some restrictions(snowplow) we are only able to get the data to ES through a NSQ channel. Is there any way to move data from Elastic Search…
INFOSYS
  • 1,465
  • 9
  • 23
  • 50
1
vote
1 answer

nsqjs clients, not receiving messages immediately from go-nsq server side

trying to learn nsq, and following the examples from here golang example and here nsqjs. I am sending messages in server side doing w/ a for loop and go routines var wg sync.WaitGroup for i := 0; i < 100; i++ { wg.Add(1) go func(x int) { …
Hokutosei
  • 2,114
  • 5
  • 24
  • 42
1
vote
1 answer

How to send the message back to the sender with messaging architecture?

I'm solving the communications between the microservices with messaging-architecture. Let's say I have a tradition application, and there're User, Post Video modules. You can create the posts, videos with it, but before that, I need to convert the…
Yami Odymel
  • 1,782
  • 3
  • 22
  • 48
1
vote
2 answers

How to connect nsqd and nsqlookupd containers?

I am trying to connect these 2 docker containers: nsqd: https://registry.hub.docker.com/u/mreiferson/nsqlookupd/ nsqlookupd: https://registry.hub.docker.com/u/mreiferson/nsqlookupd/ These are the official docker containers for nsqd and…
rexposadas
  • 3,109
  • 3
  • 27
  • 49
1
2
3 4