converting from python-kafka to confluent kafka - how to create parity with SASL_SSL, OAUTHBEARER and Tokens

Question

I have a python kafka that works and is the code:

class TokenProvider(object):
    
    def __init__(self,client_id,client_secret):
        self.client_id = client_id
        self.client_secret = client_secret
    def token(self):
        token_url = 'https://test.com/protocol/openid-connect/token'
        client = BackendApplicationClient(client_id=self.client_id)
        oauth = OAuth2Session(client=client)
        token_json = oauth.fetch_token(token_url=token_url, client_id=self.client_id, client_secret=self.client_secret)
        token = token_json['access_token']
        #print(token)
        return token

consumer = KafkaConsumer(
    group_id=None,
    bootstrap_servers=['test.com:9094'],
    security_protocol='SASL_SSL',
    sasl_mechanism='OAUTHBEARER',
    sasl_oauth_token_provider=TokenProvider(client_id,client_secret),
    ssl_check_hostname=False,
    ssl_context=create_ssl_context(),
    auto_offset_reset=offset,
    enable_auto_commit=False,
    value_deserializer=lambda m: decode(m)
    )
consumer.subscribe(topics=['test.stream'])

My confluent python is the below and I get this error

cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Property "oauthbearer_token_refresh_cb" must be set through dedicated .._set_..() function"}

c = Consumer({
    'bootstrap.servers': 'test.com:9094',
    'sasl.mechanism': 'OAUTHBEARER',
    'security.protocol': 'SASL_SSL',
    'oauthbearer_token_refresh_cb': TokenProvider(client_id,client_secret),
    'group.id': str(uuid.uuid1()),
    'auto.offset.reset': 'earliest'
})

c.subscribe(['test.stream'])

So how do I get confluent kafka to work? I appear to have an issue with oauthbearer_token_refresh_cb using OAUTHBEARER and SASL_SSL.

In essence I auth with a jwt token

score 0 · Answer 1 · answered Nov 19 '20 at 16:19

Per the documentation at https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md, the oauthbearer_token_refresh_cb option must be set using rd_kafka_conf_set_oauthbearer_token_refresh_cb(). Note, however, that you are trying to set it to a TokenProvider instance, which is not callable, so you probably want to pass TokenProvider(...).token.

SASL/OAUTHBEARER token refresh callback (set with rd_kafka_conf_set_oauthbearer_token_refresh_cb(), triggered by rd_kafka_poll(), et.al. This callback will be triggered when it is time to refresh the client's OAUTHBEARER token.

Thanks still no understanding how to implement the callback – Tampa Nov 20 '20 at 08:22 — Tampa, Nov 20 '20 at 08:22

score 0 · Answer 2 · edited Nov 23 '20 at 01:35

0

From the source, the python and go clients do not yet support oauthbearer

edited Nov 23 '20 at 01:35

OneCricketeer

179,855
19
132
245

answered Nov 22 '20 at 09:38

Tampa

75,446
119
278
425

python does not but go does – Tampa Nov 23 '20 at 13:24

converting from python-kafka to confluent kafka - how to create parity with SASL_SSL, OAUTHBEARER and Tokens

2 Answers2