How to make SASL JAAS working on Kafka on Kubernetes?

Question

I'm trying to set up Kafka on kubernetes, I setup 2 listeners and attached SASL PLAIN mechanism for both from JAAS. JAAS config loaded from k8s Secret. Somehow, its not accepting credentials and throwing authentication error, I couldn't figure out what's happening. Any input is very much appreciated.

     containers:
    - env:
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://broker-1:29092,azlbip://xxxx:9443
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS
          value: "0"
        - name: KAFKA_JMX_HOSTNAME
          value: localhost
        - name: KAFKA_JMX_PORT
          value: "9101"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: PLAINTEXT:PLAINTEXT,azlbip:SASL_PLAINTEXT
        - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
          value: "1"
        - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
          value: "1"
        - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
        - name: KAFKA_LISTENER_NAME_azlbip_PLAIN_SASL_JAAS_CONFIG
          valueFrom:
           secretKeyRef:
            name: sasl
            key: sasl
        - name: KAFKA_LISTENER_NAME_azlbip_SASL_ENABLED_MECHANISMS
          value: PLAIN
        - name: KAFKA_LISTENER_NAME_PLAINTEXT_PLAIN_SASL_JAAS_CONFIG
          valueFrom:
           secretKeyRef:
            name: sasl
            key: sasl
        - name: KAFKA_LISTENER_NAME_PLAINTEXT_SASL_ENABLED_MECHANISMS
          value: PLAIN 

Secret Content :

org.apache.kafka.common.security.plain.PlainLoginModule required username="broker" password="password" user_test="test-secret";

Finally this translates to kafka.properties on pod as

sh-4.4$ cat /etc/kafka/kafka.properties
listener.name.plaintext_plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="broker" password="password" user_test="test-secret";
jmx.port=9101
transaction.state.log.min.isr=1
group.initial.rebalance.delay.ms=0
jmx.hostname=localhost
listener.name.plaintext.sasl.enabled.mechanisms=PLAIN
advertised.listeners=PLAINTEXT://broker-1:29092,azlbip://xxxx:9443
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,azlbip:SASL_PLAINTEXT
listener.name.azlbip.sasl.enabled.mechanisms=PLAIN
broker.id=1
transaction.state.log.replication.factor=1
listeners=PLAINTEXT://0.0.0.0:29092,azlbip://0.0.0.0:9443
zookeeper.connect=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
listener.name.azlbip.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="broker" password="password" user_test="test-secret";
log.dirs=/var/lib/kafka/data
offsets.topic.replication.factor=1
sh-4.4$