Always Open Publish Channel RabbitMQ

Question

I am trying to integrate snmptrapd and RabbitMQ for delivering traps notifications to an exterior system.

My system is composed of 3 components:

A Linux virtual machine with snmptrapd and RabbitMQ (Publisher);
A Linux virtual machine with RabbitMQ (Consumer);
A Linux bare metal with docker so I can have a lot of containers sending traps (using nping)

The snmptrapd part is simple:

authCommunity execute mycom
traphandle default /root/some_script

In my first attempts the some_script was written in Python, but the performance was not perfect (20 containers sending 1 trap per second during 10 seconds, I only received 160 messages in the consumer).

#!/usr/bin/env python

import pika
import sys

message = ""

for line in sys.stdin :
        message += (line)

credentials = pika.PlainCredentials('test', 'test')

parameters  = pika.ConnectionParameters('my_ip', 5672, '/', credentials)

connection  = pika.BlockingConnection(parameters)
channel     = connection.channel()

channel.queue_declare(queue='snmp')

channel.basic_publish(exchange='',
                      routing_key='snmp',
                      body=message)
connection.close()

I switched to Perl and now I can get 200 traps/messages.

My Perl script uses Net::AMQP::RabbitMQ

#!/usr/bin/perl

use Net::AMQP::RabbitMQ;

foreach my $line ( <STDIN> ) {
    chomp( $line );
    $message = "$message\n$line";
}

my $mq = Net::AMQP::RabbitMQ->new();

$mq->connect("my_ip", {
    user     => "test",
    password => "test",
    vhost    => "/"
});

$mq->channel_open(1);
$mq->publish(1, "snmp", $message);
$mq->disconnect();

But I want better. I tried 200 containers sending 1 trap per second and it failed miserably, receiving only around 10% of messages in the consumer.

I think this has to do with the overhead of always have to open, publish and close the channel in RabbitMQ per trap received, because at the network level I receive all the messages (checked trough a tcpdump).

Is there a way to keep an always-open publish channel so I don't have to reopen/create a connection to the queue?

ikegami · Answer 1 · 2016-12-29T18:17:58.943

0

Asking if you can talk to RabbitMQ server without connecting to it first is like asking if you can talk to someone on the telephone without connecting to their phone first (by dialing and answering).

You really should reuse your connection if you're going to send multiple messages, but you do need a connection first!

Anyway, the problem isn't with the publisher. It's the consumer that's buggy if it's losing messages.

edited Dec 29 '16 at 18:17

answered Dec 29 '16 at 18:08

ikegami

367,544
15
269
518

Hi ikegami, I am questioning how can I mantain the channel always open. I'm not trying to send messages on a closed channel. Don't treat me like I am stupid. No, the problem is in the publisher. All the messages I send on the publisher arrive on the consumer. The machine that contains the publisher struggles to send all the messages, and it's not a resource problem because none of processes consumes all the CPU/Memory. – Fernando Costa Dec 30 '16 at 17:25
A channel is just a number that's sent with every message. – ikegami Dec 30 '16 at 17:53
If basic_publish was failing, you'd be getting an exception. The problem is not on the publisher. – ikegami Dec 30 '16 at 17:57
As you said "at the network level I receive all the messages". – ikegami Dec 30 '16 at 22:54

Always Open Publish Channel RabbitMQ

1 Answers1