0

Scenario: librdkafka admin API used. Each thread has it's own Producer.

Thread-A : try to create Kafka topic 'X'

Thread-B : try to create Kafka topic 'X'

Thread-C : try to create Kafka topic 'X'

as I understand 2 out of 3 threads could get error "RD_KAFKA_RESP_ERR_TOPIC_ALREADY_EXISTS = 36(TOPIC_ALREADY_EXISTS in https://kafka.apache.org/11/protocol.html)" in above attempts and I need to consider this error as a OK.

Are there any other complications due to this methodology ?

aKumara
  • 395
  • 1
  • 12
  • 1) How about catching the error? 2) You could solve this by using a distributed lock or just create the topic ahead of time. Otherwise, your app will always throw this error after the first time – OneCricketeer Sep 01 '22 at 13:58
  • plan is to check if error code returned by create topic call is RD_KAFKA_RESP_ERR_TOPIC_ALREADY_EXISTS, and if so, treat it as a creation successful. i.e. NOT treating RD_KAFKA_RESP_ERR_TOPIC_ALREADY_EXISTS as an error. – aKumara Sep 01 '22 at 14:23
  • Sure, I understand what you're asking, but I provided other options which may work better. Especially if you don't create topics dynamically at runtime, or at the very least, only use one thread / a semaphore lock for a "topic manager" outside of individual producer threads... Have you tried doing that already? What issues are you running into? – OneCricketeer Sep 01 '22 at 15:39
  • managing things at a module/object level (i.e. at individual thread) is better than trying to use locking/common manager modules. Hence, if there is are no problems in method I described, I prefer using it. Do you know any problem with my said approach ? – aKumara Sep 01 '22 at 15:49
  • The only problem I see is that you're making unnecessary API calls, IMO. Plus, you should be able to query if the topic exists before even trying to create it to prevent that error, in the first place. – OneCricketeer Sep 01 '22 at 15:55
  • "query if the topic exists before even trying to create it " this will NOT work since different threads trying to create the same topic concurrently.[i.e. with Independent module approach] – aKumara Sep 01 '22 at 16:12

0 Answers0