0

I have a python microservice application that on request sends specific message to specific kafka topic with specific principal. All parameters are user controlled in request body. How do I avoid Kafka thread safety problems?

I do know that I can control the kerberos cache via env variable KRB5CCNAME, but if I touch it, it won't be thread safe (one thread of Kafka might collide with another one afterall not letting me reach right topic with right credentials). If I do it via multiprocessing then it won't be memory efficient as I need to deal with potentially hundred of such topics. If I don't touch KRB5CCNAME then each kinit command will overwrite same cache with new ticket this way leading me to security leak (wrong app can publish to wrong topic).

I'm not in control of infrastructure and cannot opt into changing design pattern of architecture (can't get one principal with access to many topics, it has to be "bring your own credentials and topic" approach)

How to achieve concurrent Kafka sessions with different principals simultaneously from single python process without multiprocessing?

I'm using confluent-kafka-python (and accordingly librdkafka) with vanilla Flask on RHEL8 with MIT kerberos.

dimon222
  • 162
  • 2
  • 20
  • you can have more then one principle in a credential cache, and there's multiple different types of credentail cache formats. If you kinit command is removing the other credentails, I would fix that, talking to you kerberos/os team. Any solution is going to be tied to what kerberos setup you have deployed. The alternative is looking to get the krb5ccname plumbed from confluent-kafka-python -> librdkafka -> sasl->sasl-gssapi->libkrb5. – toppk Aug 18 '23 at 05:04
  • Are all credentials under the same realm? – user1686 Aug 18 '23 at 12:37
  • @toppk that's the next thing I'm researching now but it seems implementation of other types of crendetial caches isn't yet adopted fully – dimon222 Aug 18 '23 at 18:32
  • @user1686 sometimes same realm, sometimes same forest (x.y.z and a.x.y.z) – dimon222 Aug 18 '23 at 18:33

1 Answers1

1

Instead of letting libgssapi find credentials on its own, parts of the librdkafka library would need to be rewritten to explicitly acquire Kerberos credentials using the GSSAPI Credential Store extensions now available in MIT libkrb5 (and I believe in Heimdal as well).

The gssproxy daemon and the Apache mod_auth_gssapi module are good examples of using these extensions in C code to access multiple sets of credentials within the same process. For pure Python code, python-gssapi makes these extensions easy to use via gssapi.Credential – ideally the Python kafka bindings would use that somehow.

But other than actually rewriting the parts of the library that read process-global environment variables to Not Do That, there's no secret backdoor to make a non-threadsafe library somehow thread-safe. (Except for, perhaps, separating the Kafka-related code into its own daemon and ensuring that it would fork a new process for every topic – it shouldn't be horribly inefficient, as 'multiprocessing' appears to use fork() instead of spawning a whole new process, i.e. much of the memory is actually shared copy-on-write.)

If I don't touch KRB5CCNAME then each kinit command will overwrite same cache with new ticket this way leading me to security leak (wrong app can publish to wrong topic).

This is not necessarily true with current MIT Kerberos or Heimdal Kerberos. While the FILE:-based caches can only hold tickets for a single principal, both libraries also support DIR:<path> for directory-based cache collections, where each new kinit only adds a new ticket cache (and klist -A would show all). I think Heimdal's SQLITE: is multi-principal as well.

While DIR still has a concept of "active" cache for legacy APIs, a standard request for a service principal ticket will use all of them – not just the active one. With MIT Kerberos, the algorithm for choosing between caches in a collection can be influenced by ~/.k5identity to map service principal domains to Kerberos realms.

Unfortunately, this will be completely useless in your case, as it sounds like all credentials are used for the exact same server.

user1686
  • 13,155
  • 2
  • 35
  • 54
  • Thank you for your clarification. I did already have exactly this trick with `python-gssapi` but got immediately disappointed to find out that it's impossible to get it working with `confluent-kafka-python` since it doesn't offer a way to use its instances. – dimon222 Aug 18 '23 at 18:37