0

I am trying to write functional test using embedded kafka. but getting below error while starting cluster:

An exception or error caused a run to abort: org.apache.kafka.common.config.ConfigDef$ValidString.(Ljava/util/List;Lorg/apache/kafka/common/config/ConfigDef$1;)V java.lang.NoSuchMethodError: org.apache.kafka.common.config.ConfigDef$ValidString.(Ljava/util/List;Lorg/apache/kafka/common/config/ConfigDef$1;)V at org.apache.kafka.common.config.ConfigDef$ValidList.(ConfigDef.java:895)`

My Pom.xml has this dependancies 

    `
        <dependency>
        <groupId>net.manub</groupId>
        <artifactId>scalatest-embedded-kafka_2.11</artifactId>
        <version>0.10.0</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10-assembly_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>2.1.0</version>
        <classifier>test</classifier>
        <scope>test</scope>
    </dependency>`
Simas Joneliunas
  • 2,890
  • 20
  • 28
  • 35
p1q1w1e1
  • 1
  • 1

2 Answers2

0

You're missing the clients

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>2.1.0</version>
    </dependency>
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Its functional test for spark job. which is basically reading data from s3 and passing to kafka. So in functional test I am trying to start embedded kafka which is used for testing purpose. And test is failing while starting that kafka. – p1q1w1e1 Feb 05 '20 at 16:57
  • `implicit val config: EmbeddedKafkaConfig = EmbeddedKafkaConfig(kafkaPort = 9092, zooKeeperPort = 2182) val topic: String = "test-topic" var appExecutorThread: ExecutorService = _ before { SparkSession.clearActiveSession() EmbeddedKafka.stop() while (EmbeddedKafka.isRunning) Thread.sleep(100) EmbeddedKafka.start()(config) EmbeddedKafka.createCustomTopic(topic)(config) while (!EmbeddedKafka.isRunning) Thread.sleep(100) appExecutorThread = Executors.newSingleThreadExecutor() } `` – p1q1w1e1 Feb 05 '20 at 16:57
0

Please see the build.sbt or pom.xml for scalatest-embedded-kafka 0.10.0. Both are using kafkaVersion = "0.10.1.0".

You can see in the files linked, that kafka_2.11 dependency is not with scope provided, so you probably don't need to include the kafka_2.11 dependency at all. If you do include it for some reason, you generally want to ensure that the class signatures needed in the transitive dependency are all present. The README file (same repo using the build.sbt link above) confirms that you probably don't need the kafka dependency. Your build tool should automatically bring in any transitive dependencies. I would try removing it, and or changing the version if that doesn't fix it.

ELinda
  • 2,658
  • 1
  • 10
  • 9