Ruby nsq how to listen for new messages

Question

My setup is as follows: 1 Microservice that receives a request and writes a message in the queue(nsq) and a second microservice that must read the messages in the queue and do something based on them.

I am new to the nsq concept in ruby on rails. I have installed the nsq from here: http://nsq.io/overview/quick_start.html. I have also used this gem to facilitate pushing messages: https://github.com/wistia/nsq-ruby.

I have been able to queue the message. This part was easy enough.

Question:

How do I always listen in the 2nd microservice to figure out when something was pushed so I can consume it?

This is how I push messages:

require 'nsq'

class NsqService
  attr_accessor :producer, :topic

  def initialize(topic)
    @topic = topic
    @producer = producer
  end

  def publish(message)
    producer.write(message)
  end

  private

  def producer(nsqd='127.0.0.1:4150')
    Nsq::Producer.new(
      nsqd: nsqd,
      topic: @topic
    )
  end
end

It looks like you need a consumer. See documentation https://github.com/wistia/nsq-ruby#consume-messages — lacostenycoder, Mar 17 '18 at 14:52
@lacostenycoder yes, I do need a consumer. The problem I am having a though time with is when to activate that consumer. What even should trigger the consumer to come online ? Is the consumer triggered whenever I push a message? How does it always listen? These are my questions. — Lucian Tarna, Mar 17 '18 at 14:54
Rails is a restful web server meaning it only listens for GET and POST requests on whatever port your running on (i.e. `http://localhost:3000` in development). In order to use nsq you'll likely need a seperate server which should be pretty straightforward to write according to the documentation on the gem. However depending on what version of Rails you're using, you might prefer to just user ActionCable https://github.com/rails/rails/tree/master/actioncable — lacostenycoder, Mar 17 '18 at 15:09
sorry @lacostenycoder I still don't understand. What questions are in my head still unanswered: 1. Are you suggesting to always keep a connection open between my applications (with action-cable) and when I push something to the queue to push a message through he action-cable command to make the other app read? 2. How to know when to activate the consumer to read? — Lucian Tarna, Mar 17 '18 at 15:30
I was suggesting to that you might just use actioncable instead of nsq as it's built into rails and you won't need to run/build a seperate server. But I guess the solution depends on exactly what your use case is. Again, refer to the documentation. — lacostenycoder, Mar 17 '18 at 16:23
@lacostenycoder oh, I can't nsq is a requirement. I did read the documentation, and searched on the net. There is no pattern given anywhere to follow when it comes to consumer pooling the messaging queue. I can do it with a cron for example, i can do it with actioncable, I can do it with an infinite loop... that's my problem, I want a pattern — Lucian Tarna, Mar 17 '18 at 16:35
Did you have a look at http://nsq.io/deployment/topology_patterns.html ? — lacostenycoder, Mar 17 '18 at 17:46
@lacostenycoder da, but then in the gem it says to clean up after myself(close the connection) and I said I don't understand. Connection != event observer ? If it is so, this answers my question and you should really put it as an answer and thank you very much anyway for giving me so much of your time — Lucian Tarna, Mar 17 '18 at 18:35

score 1 · Accepted Answer · answered Mar 17 '18 at 21:13

Example code on the nsq-ruby gem give the following code example:

    require 'nsq'
    consumer = Nsq::Consumer.new(
      nsqlookupd: '127.0.0.1:4161',
      topic: 'some-topic',
      channel: 'some-channel'
    )

    # Pop a message off the queue
    msg = consumer.pop
    puts msg.body
    msg.finish

    # Close the connections
    consumer.terminate

You could wrap this in a class for your service etc. You'll likely need to run some kind of middleware or separate processes to handle these connections. If there are parts of your rails code which need to interface with NSQ. While I have not used NSQ, I've used Sidekiq for background jobs and running async processes which has good instructions and examples of how to configure those middleware. For more suggestions and help, you might try contacting some of the maintainers of the ruby gem. I'm sure they can point you in the right direction.

Ruby nsq how to listen for new messages

1 Answers1