1

I am using syslog-ng(latest version 3.30) and kafka(2.2)

  1. syslog-ng and kafka is running in 3 nodes
  2. Client send logs to multiple syslog-ng nodes
  3. syslog-ng nodes send these logs to kafka broker
  4. For kafka syslog-ng is producer

The problem i am facing is that multiple syslog-ng who act as a producer are sending the same data to the topic. In kafka topic if one client is sending message like ABC to multiple syslog-ng then all the nodes of syslog-ng receive the same message ABC and producers(multiple syslog-ng nodes) send ABC message to the kafka so there will 3 ABC message


@define kafka-implementation kafka-c

#############################################
source s_network_udp {
     network(
     transport("udp")
     port(514)
);
};

#############################################

rewrite test {
set(
"test",
        value("tags")
);
};

#############################################
destination pfsense_kafka
{
kafka (
bootstrap-servers("xx.xx.xx.xx:9092")
topic("logs")
properties_file("/etc/syslog-ng/syslog_producer.properties")
 message("$(format-json time=$ISODATE tags host=$HOST message='${MSGHDR}${MSG}')")
);
};

log{ source(s_network_udp); rewrite(test); destination(pfsense_kafka);};

syslog_producer.properties

acks=all
#transactional.id=pfsenseProducer
#transaction.state.log.replication.factor=1
enable.idempotence=true
#retries=10000000
message.send.max.retries=10000000
max.in.flight.requests.per.connection=1

I have read about Idempotence Producer and setting transaction.id will provide exactly once delivery,but nothing work for me

when i enable enable.idempotence=true,message.send.max.retries=10000000,max.in.flight.requests.per.connection=1 I receive the messages sent by syslog-ng but when i enable 'transaction.id' i didn't receive any messages

aniketpant
  • 113
  • 1
  • 8

1 Answers1

0

Let's start with the idempotent producer. By setting enable.idempotence=true you are set. The transaction.id is not needed, in fact you should not set that. This is the good news.

The bad news is, that the idempotent producer does provide you exactly one delivery. But that means that if you provide a message A, that is delivered once to kafka and not less or more.

If you have two producer (for example two syslog-ng) and they both send message to kafka via the producer, those message (even if the content is the same) are not the same message (message A and B, but content is C). The result is that both sends message A and B, and the kafka topic going to look like C, C. Exactly not what you want. But something that

Delivering exactly one C would required application level logic.

kokan
  • 64
  • 3