So for some reason I can't get my connections just right with MSK via the Kafka Java API. I can get producers/consumers to work with MSK using conduktor and Kafka CLI tools. However when I try to hook up my Scala code I can't get it to work. So I am using the config as follows to connect via conduktor and Kafka CLI tools.
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandle
and for my Scala application I am setting up producers/consumers using a similar pattern
def props: Properties = {
val p = new Properties()
....
p.setProperty("security.protocol", "SASL_SSL")
p.setProperty("sasl.mechanism", "AWS_MSK_IAM")
p.setProperty("sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;")
p.setProperty("sasl.client.callback.handler.class", "software.amazon.msk.auth.iam.IAMClientCallbackHandler")
p
}
val PRODUCER = new KafkaConsumer[AnyRef, AnyRef](props)
So the code works when I omit the security config lines and run against a local instance of Kafka but when I try to hit the MSK it seems like it isn't constructing a consumer and I get the following error.
java.lang.IllegalStateException: You can only check the position for partitions assigned to this consumer.
However, the locally running instance works. So this makes me think I'm not setting up something correctly in the config to connect to the MSK.
I am trying to follow the following tutorial and I am using Scala 2.11 and Kafka versions 2.41. I also added the aws-msk-iam-auth to my build.sbt (1.1.0). Any thoughts or solutions?
