Disable Direct Runner Logs in Apache Beam for Kafka Consumer

Question

Seen a similar question asked, but on dataflow logging and not direct logging.

Basically, I want to turn off the wave of KafkaIO read (consumer) logs. I have tried setting the logging levels in SDK harness as follows.

var kafkasLogs =
        SdkHarnessOptions.SdkHarnessLogLevelOverrides.from(
            new HashMap<>(
                Map.of(
                    "org.apache.kafka.clients.consumer.internals.SubscriptionState",
                    SdkHarnessOptions.LogLevel.ERROR.name())));
    options.setSdkHarnessLogLevelOverrides(kafkasLogs); // extends sdkharnessoptions

I have also tried variations of the above but have not been successful in my bid to silence the consumer logs.

What is the way I can shut these logs off without messing with the additional logging of my pipeline?

Alberto Serna · Answer 1 · 2023-03-15T12:21:16.520

SOLUTION 1: I made it work with org.apache.log4j.Logger

   import org.apache.log4j.Logger
//import org.slf4j.{Logger, LoggerFactory}

object KafkaConsumer {

  // NOTE 1: I did not have to set up the below JVM flags in the end
  // (it was only filtering out the logs from the app not the internal beam ones)
  // -Dlog4j.configurationFile=src/main/resources/log4j2.properties
  // -Djava.util.logging.config.file=src/main/resources/log4j2.properties
  //  --beam.log4j2ConfigurationFile=src/main/resources/log4j2.properties
  val log = Logger.getLogger(this.getClass.getName)

// NOTE 2: it did not work out either
// --sdkHarnessLogLevelOverrides={\"org.apache.kafka.clients.consumer.internals\":\"ERROR\"}

// NOTE 3: I have not been able to filter out using slf4j
//  val log: Logger = LoggerFactory getLogger getClass.getName

  def main(cmdlineArgs: Array[String]): Unit = {
    import org.apache.logging.log4j.core.config.Configurator
    Configurator.initialize(null, "log4j2.xml")
    log.info("Init SCIO Context...")

being log4j2.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="org.apache.kafka.clients.consumer.internals.SubscriptionState" level="ERROR"/>
        <Root level="INFO">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>

and build.sbt (I am using SCIO) and I had multiple slfj4 bindings

libraryDependencies ++= Seq(
      "com.spotify" %% "scio-extra" % scioVersion,
      "com.spotify" %% "scio-avro" % scioVersion,
      "com.spotify" %% "scio-core" % scioVersion,
      "com.spotify" %% "scio-test" % scioVersion % Test,
      "com.spotify" %% "scio-google-cloud-platform" % scioVersion,
      "org.apache.beam" % "beam-runners-direct-java" % beamVersion,
      "org.apache.beam" % "beam-runners-google-cloud-dataflow-java" % beamVersion,
//      "org.slf4j" % "slf4j-simple" % "1.7.36",
//      "org.slf4j" % "slf4j-api" % "1.7.36",
      "com.google.cloud" % "google-cloud-bigquery" % "2.13.2",
      "com.google.cloud" % "google-cloud-storage" % "1.17.0",
      "org.mockito" %% "mockito-scala" % "1.16.34" % Test,
      "org.mockito" %% "mockito-scala-scalatest" % "1.16.15" % Test,
      "org.mockito" %% "mockito-scala-specs2" % "1.16.15" % Test,
      "org.mockito" %% "mockito-scala-cats" % "1.16.15" % Test,
      "org.mockito" %% "mockito-scala-scalaz" % "1.16.15" % Test,
      "jp.ne.opt" %% "bigquery-fake-client" % "0.1.0" % Test,
      ("com.my.jar" % myJar" % jarVersion)
        .exclude("com.twitter", "chill_2.11")
        .exclude("org.scala-lang.modules", "scala-xml_2.11")
        .exclude("com.fasterxml.jackson.module", "jackson-module-scala_2.11")
        .exclude("org.scala-lang.modules", "scala-parser-combinators_2.11")
        .exclude("org.slf4j", "slf4j-log4j12")
    )

with "-Dlog4j.configurationFile=src/main/resources/log4j2.xml" and these dependencies should suffice too:

"org.slf4j" % "slf4j-simple" % "1.7.36",
"org.apache.logging.log4j" % "log4j-core" % "2.19.0",
"org.apache.logging.log4j" % "log4j-api" % "2.19.0",
 "org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.19.0",

SOLUTION 2:

with JVM -Dlog4j.configurationFile=src/main/resources/log4j2.xml

and

import org.slf4j.{Logger, LoggerFactory}

class MyClass[T] {
  val log: Logger = LoggerFactory getLogger getClass.getName

after: after_Configurator_initialize

before: before_Configurator_initialize

Disable Direct Runner Logs in Apache Beam for Kafka Consumer

1 Answers1