4

I am using spark-streaming-kafka-0-10_2.10 of version 2.0.2 for spark streaming job. I got warns like this: 17/10/10 16:42:25 WARN KafkaUtils: overriding enable.auto.commit to false for executor 17/10/10 16:42:25 WARN KafkaUtils: overriding auto.offset.reset to none for executor 17/10/10 16:42:25 WARN KafkaUtils: overriding executor group.id to spark-executor-dump_user_profile 17/10/10 16:42:25 WARN KafkaUtils: overriding receive.buffer.bytes to 65536 see KAFKA-3135

AND when i look at the source code, there is a piece of code fixed the params named fixKafkaParams(...) in KafkaUtils shown as below:

```

logWarning(s"overriding ${ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG} to false for executor")
kafkaParams.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false: java.lang.Boolean)

logWarning(s"overriding ${ConsumerConfig.AUTO_OFFSET_RESET_CONFIG} to none for executor")
kafkaParams.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none")

// driver and executor should be in different consumer groups
val originalGroupId = kafkaParams.get(ConsumerConfig.GROUP_ID_CONFIG)
if (null == originalGroupId) {
  logError(s"${ConsumerConfig.GROUP_ID_CONFIG} is null, you should probably set it")
}
val groupId = "spark-executor-" + originalGroupId
logWarning(s"overriding executor ${ConsumerConfig.GROUP_ID_CONFIG} to ${groupId}")
kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG, groupId)

// possible workaround for KAFKA-3135
val rbb = kafkaParams.get(ConsumerConfig.RECEIVE_BUFFER_CONFIG)
if (null == rbb || rbb.asInstanceOf[java.lang.Integer] < 65536) {
  logWarning(s"overriding ${ConsumerConfig.RECEIVE_BUFFER_CONFIG} to 65536 see KAFKA-3135")
  kafkaParams.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 65536: java.lang.Integer)
}

} ``` How can i get through this? thanks very much

Ruslan Ostafiichuk
  • 4,422
  • 6
  • 30
  • 35
deerluffy
  • 41
  • 4

1 Answers1

0

"KafkaUtils: overriding auto.offset.reset to none for executor" is the usual behavior in KafkaUtils and this does not create any issue. This can be ignored.

This is written in this way only in KafkaUtils in order to Tweak kafka params to prevent issues on executors however you can check that on driver, it does not change the value of auto.offset.reset and keeps the value which you define in your kafkaParams. Below is the link of kafkaUtils for the reference.

https://github.com/apache/spark/blob/master/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala

Initial I even thought that this might be issue but after the execution of my kafka code, I did not face any issue.

jin
  • 11
  • 4