0

I need to add partitions to an existing Kafka topic. I'm aware that it is possible to use the /bin/kafka-topics.sh script to achieve this, but I would prefer to do this through the Confluent REST api.

As far as I see there is no documented endpoint in the api reference, but I wonder if someone else here was able to make this work.

Edit: As it does seem to be impossible to use the REST api here, I wonder what the best practice is for adding partitions to an existing topic in a containerized setup. E.g. if there is a custom partioning scheme that maps customer ids to specific partitions. In this case the app container would need to adjust the partition count of the kafka container.

code-gorilla
  • 2,231
  • 1
  • 6
  • 21

2 Answers2

0

The Confluent REST Proxy has no such endpoint for topic update administration.

You would need to use the shell script or the corresponding AdminClient class that the shell script uses

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks. Do you know what the best practice is for a containierized setup, where the app can neither use the scripts nor call the classes? The only solution I see is to deploy a small web service with the kafka container that calls the script internally. – code-gorilla May 15 '22 at 21:39
  • Not sure what you mean by "cannot call the classes". You would just need to import the Kafka client library, but that would need to be REST service to use it repeatedly. Or you can simply `docker run` (or kubectl exec for k8s) the scripts as a standalone container – OneCricketeer May 16 '22 at 00:18
  • Terraform providers and k8s operators also exist for Kafka topics – OneCricketeer May 16 '22 at 00:21
0

This is the solution I ended up with:

  • Create a small http service that is is deployed within the kafka docker image
  • The http service accepts requests to increase the partition count and directs the requests to the kafka admin scripts (bin/kafka/kafka-topics.sh)
    • Something similar could have been achieved by using the AdminClient NewPartitions api in the Java Kafka lib. This solution has the advantage that the kafka docker image does not have to be changed, because the AdminClient can connect through network from another container.

For a production setup the AdminClient is preferrable, I decided for the integrated script approach because of the chosen language (rust).

code-gorilla
  • 2,231
  • 1
  • 6
  • 21