PyFlink 1.11 not connecting to Confluent Cloud (Kafka) cluster

Question

I am configuring PyFlink to connect to a confluent cloud kafka cluster. I am using SASL/PLAIN. Below is the code snippet:

""" CREATE TABLE {0} (
                `transaction_amt` BIGINT NOT NULL,
                `event_id` VARCHAR(64) NOT NULL,
                `event_time` TIMESTAMP(6) NOT NULL
              )
              WITH (
                'connector' = 'kafka',
                'topic' = '{1}',
                'properties.bootstrap.servers' = '{2}',
                'properties.group.id' = 'testGroupTFI',
                'format' = 'json',
                'json.timestamp-format.standard' = 'ISO-8601',
                'properties.security.protocol' = 'SASL_SSL',
                'properties.sasl.mechanism' = 'PLAIN',
                'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username=\"{3}\" password=\"{4}\";'
            ) """.format(table_name, stream_name, broker, user, secret)

I am getting this error:

{
"applicationARN": "arn:aws:kinesisanalytics:us-east-2:xxxxxxxxxxx:application/sentiment",
"applicationVersionId": "13",
"locationInformation": "org.apache.flink.runtime.taskmanager.Task.transitionState(Task.java:973)",
"logger": "org.apache.flink.runtime.taskmanager.Task",
"message": "Source: TableSourceScan(table=[[default_catalog, default_database, input_table]], fields=[transaction_amt, event_id, event_time]) -> Sink: Sink(table=[default_catalog.default_database.output_table_msk], fields=[transaction_amt, event_id, event_time]) (3/12) (25a905455865731943be6aa60927a49c) switched from RUNNING to FAILED.",
"messageSchemaVersion": "1",
"messageType": "WARN",
"threadName": "Source: TableSourceScan(table=[[default_catalog, default_database, input_table]], fields=[transaction_amt, event_id, event_time]) -> Sink: Sink(table=[default_catalog.default_database.output_table_msk], fields=[transaction_amt, event_id, event_time]) (3/12)",
"throwableInformation": "org.apache.flink.kafka.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka producer\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:432)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)\n\tat org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.<init>(FlinkKafkaInternalProducer.java:78)\n\tat org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.createProducer(FlinkKafkaProducer.java:1141)\n\tat org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initProducer(FlinkKafkaProducer.java:1242)\n\tat org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initNonTransactionalProducer(FlinkKafkaProducer.java:1238)\n\tat org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:940)\n\tat org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.beginTransaction(FlinkKafkaProducer.java:99)\n\tat org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:398)\n\tat org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:389)\n\tat org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1111)\n\tat org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)\n\tat org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)\n\tat org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)\n\tat org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)\n\tat org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)\n\tat org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:474)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:470)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:529)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:724)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:549)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.flink.kafka.shaded.org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for org.apache.kafka.common.security.plain.PlainLoginModule\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:158)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:450)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:421)\n\t... 23 more\nCaused by: javax.security.auth.login.LoginException: No LoginModule found for org.apache.kafka.common.security.plain.PlainLoginModule\n\tat java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:731)\n\tat java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)\n\tat java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)\n\tat java.base/java.security.AccessController.doPrivileged(Native Method)\n\tat java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)\n\tat java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:581)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)\n\tat org.apache.flink.kafka.shaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:147)\n\t... 28 more\n"

}

I feel that SASL is not supported by PyFlink SQL Connector for 1.11 or 1.13. Is this is correct? Is there a workaround I can work on?

PyFlink 1.11 not connecting to Confluent Cloud (Kafka) cluster

0 Answers0