0

I understand that I can send log messages from applications which use log4j to Apache Kafka by using the log4j kafka appender. For example,

log4j.appender.KAFKA_HIVE_AUDIT=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
log4j.appender.KAFKA_HIVE_AUDIT.Topic=hive_audit_log
log4j.appender.KAFKA_HIVE_AUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n

This is described here: how to use Kafka 0.8 Log4j appender

I'm running Kafka 0.9 which has kerberos enabled. Is there a way to have the log4j appender perform authentication to Kafka? Some kind of service account?

Is there a way to for the machine which the java program is running on to authenticate to the Kafka cluster via kerberos before running the log4j appender?

If that doesn't work, is there a way to grant write privileges to unauthenticated producers on kerberized Kafka by machine? (And still require kerberos authentication for consumers)?

Community
  • 1
  • 1
leontp587
  • 791
  • 2
  • 9
  • 21

2 Answers2

1

If you look at the Kafka 0.9 source, you will see that Kerberos authentication is not supported by the KafkaLog4jAppender, despite the fact that it was added for Kafka producers and consumers generally.

Kerberos support was only added to the KafkaLog4jAppender in version 0.10.

From a quick glance at the code, it looks like it would be straightforward to create a custom appender that extends the KafkaLog4jAppender and adds the necessary bits for Kerberos support.

Todd Gibson
  • 1,005
  • 7
  • 16
-3

Kafka is Open Source, why don't you just check the code and work out which properties are related to Kerberos and how they are used?

A quick look at "trunk" i.e. https://github.com/apache/kafka/blob/trunk/log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java hints that you must attach a JAAS config file, and specify which "context" to use in that file, so that the Kafka client can use the standard Java Security libraries. And optionally provide a custom Kerberos config file if it is not in the standard location (i.e. /etc/krb5.conf).

So in the end it's a matter of understanding JAAS -- and finding out which Kafka and/or Log4J properties to set.

And with Google, a tutorial is always a few clicks away -- e.g. that kerberized_kafka post.

Samson Scharfrichter
  • 8,884
  • 1
  • 17
  • 36