1

I want to stream a kafka topic from Glue job but I got the following error

StreamingQueryException: Not authorized to access topics: [topic_name]

This my current script

# Script generated for node Kafka Stream
dataframe_KafkaStream_node1 = glueContext.create_data_frame.from_options(
    connection_type="kafka",
    connection_options={
        "connectionName": "corp_cli_kafka_connection",
        "classification": "json",
        "startingOffsets": "latest",
        "topicName": "topic_name",
        "inferSchema": "true",
        "typeOfData": "kafka"
    },
    transformation_ctx="dataframe_KafkaStream_node1",
)

How to set the SASL JAAS config ?

"kafka.bootstrap.servers": "server_url:9092",
"kafka.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=XXXXXXXXXX password=XXXXXXXXXXXX;",
"kafka.sasl.mechanism": "PLAIN",
"kafka.security.protocol": "SASL_SSL"
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Smaillns
  • 2,540
  • 1
  • 28
  • 40
  • The issue is here is not with Authentication and related to Authorization which indicates the credentials you are using doesn't have access to topic that you are trying to access. – Prabhakar Reddy Aug 13 '22 at 02:17
  • But we need to add the JAAS config to the connection_options ! – Smaillns Aug 14 '22 at 11:12
  • @Smaillns did you ever figure this out? – Nadim Younes Aug 24 '23 at 02:28
  • @NadimYounes you need a valid connection between your aws accnt and kafka cluster, for the kafka config you can set it using pyspark as the following – Smaillns Aug 24 '23 at 08:22
  • df.selectExpr( "CAST(id AS STRING) AS key", "to_json(struct(metadata,payload)) AS value" ).write.format("kafka").option("topic", topic).option( "kafka.ssl.endpoint.identification.algorithm", "https" ).option("kafka.bootstrap.servers", bootstrap_server) .option( "kafka.sasl.jaas.config", f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{username}" password="{password}";', ).option( "kafka.sasl.mechanism", "PLAIN" ).option( "kafka.security.protocol", "SASL_SSL" ).mode("append").save() – Smaillns Aug 24 '23 at 08:24

0 Answers0