-1

In the test environment, there are rabbit consumers connected to the RELAY-* queues even when the app is stopped. This is causing messages to the queues being consumed by these “ghost” consumers, and the actual application receiving no data unless the application’s consumer happens to pick it up first.

For example, rabbitTST queue RELAY-TASK-PUBLISH-QUESTION-STORE RELAY (grails app) should be the only consumer, it is shut down at the moment on the server and yet there are currently 14 consumers.

Not a rabbit/queue/consumer expert but I don’t think that is how its supposed to work.

Version: RabbitMQ 3.3.5, Erlang R14B04

I could see ghost consumers on management plugin. There should only be 2 consumers but current status keeps changing. Sometimes it is 14 consumers, sometimes 10, likewise.

The app was running on a tomcat instance. When stopped the app is shut down on that instance. Certificate is valid until 2038.

"=ERROR REPORT==== 5-Oct-2017::17:04:14 ===
error on AMQP connection <0.13009.515>:
{ssl_upgrade_error,ekeyfile}
=INFO REPORT==== 5-Oct-2017::17:04:14 ===
accepting AMQP connection <0.13019.515> (10.142.33.2:22343 -> 10.142.18.21:5671)
=ERROR REPORT==== 5-Oct-2017::17:04:14 ===
SSL: 1112: error:[] /etc/rabbitmq/ssl/rabbitmq_wildcard.key
[{ssl_connection,init_private_key,5},
{ssl_connection,ssl_init,2},
{ssl_connection,init,1},
{gen_fsm,init_it,6},
{proc_lib,init_p_do_apply,3}]" `
Adam Kotwasinski
  • 4,377
  • 3
  • 17
  • 40
Shweta Lokhande
  • 39
  • 7

1 Answers1

0

You can use command line operation to find the consumers.

First, lets find which channels actually have these consumers:

rabbitmqctl list_channels pid name connection consumer_count

This way you should be capable of identifying which channels have these "ghost" consumers (as they will have consumer_count > 0).

Now, let's find the connections:

rabbitmqctl list_connections pid name peer_host peer_port connected_at user vhost | grep ${channel}

This way you should identify the host/port that started the connection.

RabbitMQ's management API on REST exposes some very similar capabilities. You can even go to management console, pick the problematic queue, and then expand the "consumers" section. Then you can find the channel, and ultimately, connection.

If the connection is stuck for some reason on RabbitMQ server side, you can use 

rabbitmqctl close_connection <connectionpid> <reason>

or close it from UI.

Adam Kotwasinski
  • 4,377
  • 3
  • 17
  • 40
  • Hi Adam Thank you so much for the help. Just wanted to confirm in case of Ghost consumers the consumer_count > 0 or consumer_count > 1 ? – Shweta Lokhande Oct 09 '17 at 14:44
  • I am using RabbitMQ 3.3.5 Also I am getting error for the below command rabbitmqctl list_connections pid name peer_host peer_port user vhost connected_at Listing connections ... Error: {bad_argument,connected_at} – Shweta Lokhande Oct 09 '17 at 16:25
  • Remove the `connected_at` then, it was just a timestamp. Regarding `>0` or `>1`, this depends on your application - you need to find something that looks weird to you. If you have (for example) only 1 consumer per connection, and you see 2 or more, then try shutting it down, and see where the failures appear - that should point you to the place that is creating these "ghost" consumers. – Adam Kotwasinski Oct 09 '17 at 17:19
  • Hi Adam I couldn't grep the channels. No result found. I could see some random channels getting added in the consumers section of the queue on management plugin. I was referring them as the ghost consumers of the queue. Will closing the connections help ? – Shweta Lokhande Oct 09 '17 at 18:34
  • Closing connections should cause some exceptions in consumers, so that might help you identify them. But take a look into management UI - you should be capable of clicking through `queue -> consumer -> channel -> connection` to find out who creates that ghost consumer. – Adam Kotwasinski Oct 09 '17 at 18:59
  • I am a little bit taken aback as I am unable to comprehend the situation that by clicking on 'connections', it is pointing to other rabbit nodes. We have a cluster of 3 rabbit nodes. The data from Rabbitmq should be consumed by app running on tomcat server instance. – Shweta Lokhande Oct 09 '17 at 19:56