0

Trying to use Flink 1.9 SQL-Client with Kafka without success. After figuring out about the required jar files and copying them into the lib directory, I am getting the following run-time exception when doing SELECT * FROM table-name:

Flink SQL> select * from default_catalog.default_database.member_customer_newsletters ;
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.ConsumerRecord

Doing jar -tf of the jar-file, I can see the class ConsumerRecord being there:

jar -tf ./lib/flink-sql-connector-kafka-0.11_2.12-1.9.0.jar|grep 'ConsumerRecord'
org/apache/flink/kafka011/shaded/org/apache/kafka/clients/consumer/ConsumerRecord.class

So, I am not sure why it is trhowing a ClassNotFoundException as the class is already in the jar-file?

I just need to add the run-time is looking for "org.apache.kafka.clients.consumer.ConsumerRecord" but because this jar-file is shaded the full qualified name for the class is "org.apache.flink.kafka011.shaded.org.apache.kafka.clients.consumer.ConsumerRecord.class"

But, this should be true with any other class in this shaded jar as well!

Behzad Pirvali
  • 764
  • 3
  • 10
  • 28

1 Answers1

0

There are two different classes:

org.apache.kafka.clients.consumer.ConsumerRecord
org.apache.flink.kafka011.shaded.org.apache.kafka.clients.consumer.ConsumerRecord

In flink-sql-connector-kafka-0.11_2.12-1.9.0.jar, you found the class

org.apache.flink.kafka011.shaded.org.apache.kafka.clients.consumer.ConsumerRecord

while Flink is complaining about:

org.apache.kafka.clients.consumer.ConsumerRecord

The first is a class used internally by Flink, after a kind of copy-paste from Kafka.

The second one is a class in kafka-clients-0.11.0.2.jar.

So Flink is right to complain about a missing library.

Guillaume Vauvert
  • 441
  • 6
  • 15
  • Sorry for the late response, was on vacation. I also did copy your suggested jar into the lib folder before, but same error. Flink 1.7.2 works without your suggested jar. I also thought that this jar should solve the issue, but it does not – Behzad Pirvali Sep 30 '19 at 17:13
  • My answer was to explain that Flink is right to complain. It was to explain how to fix the issue. I have updated my answer. – Guillaume Vauvert Oct 17 '19 at 07:07
  • The explanation makes sense of course, but what is the solution for it? As mentioned above, I have copied this jar into the lib folder and I am still getting the exact same error. Also, Flink 1.7.2 works without this jar. – Behzad Pirvali Oct 18 '19 at 19:02