We are using log4j2 and kafkaAppender to send logs to a topic which is consumed by Splunk. Kafka topic has 5 partitions and 24 hour retention. All our services(~ 14) send log events to the same topic.
Below is our log4j configuration:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug">
<Appenders>
<Console name="consolelog" target="SYSTEM_OUT">
<PatternLayout pattern="" chatset="UTF-8"/>
</Console>
<Kafka name="kafka" topic="topic1" ignoreExceptions="false" syncSend="false">
<PatternLayout pattern="" chatset="UTF-8"/>
<Property name="bootstarp.servers">hostname:port</Property>
<Property name="security.protocol">SSL</Property>
<Property name="ssl.truststore.location">abc.jks</Property>
<Property name="ssl.truststore.password">abcd</Property>
<Property name="ssl.keystore.location">def.jks</Property>
<Property name="ssl.keystore.password">defg</Property>
<Property name="ssl.key.password">defg</Property>
</Kafka>
<Async name="Async">
<AppenderRef ref="kafka"/>
</Async>
<Async name="console-log">
<AppenderRef ref="consolelog"/>
</Async>
</Appenders>
<Loggers>
<AsyncLogger name="com.abc" level="debug" addivity="false">
<AppenderRef ref="Async"/>
</AsyncLogger>
<AsyncLogger name="org.springframework" level=debug"" addivity="false">
<AppenderRef ref="Async" level="ERROR"/>
<AppenderRef ref="console-log" level="ERROR"/>
</AsyncLogger>
<AsyncLogger name="com.def" level="warn" addivity="false">
<AppenderRef ref="Async"/>
</AsyncLogger>
<Root level="info">
<AppenderRef ref="Async"/>
</Root>
<Logger name="org.apache.kafka" level="WARN"/>
</Loggers>
</Configuration>
We are seeing duplicate events in Splunk.
Splunk team has informed us that they are seeing same event across different partitions having different offsets.
Is it possible that the above log4j configuration might be causing this issue? How should I troubleshoot this further? Have any of you ever faced such an issue with similar setup and what was the root cause, resolution in your case?