3

I want to run multiple Kafka brokers on Mesos as Marathon jobs. I'm thinking what to use as a broker ID. I know that some people are using IP based broker ID, but I read that this approach is not optimal for situations when migrating the brokers to different machines. Any insights?

Related to that, is there a way to do a rolling deploy of Kafka brokers in a way that Marathon waits for the old broker instances to replicate the data to the new ones before killing them.

Domen P
  • 163
  • 10

1 Answers1

2

I created a Docker image which can start on Marathon and also scale up and down. To get a unique broker id, I concatenate the last octet of the IP address and the port number which Marathon devises to the Docker container.

I understood that this shouldn't be a problem, because normally the clients connect via ZooKeeper, which itself has the nodes and their connection info, so you shouldn't need to take care of the details. Or I misunderstood the question...

Have a look at

Tobi
  • 31,405
  • 8
  • 58
  • 90
  • I'm asking about rolling deploy because I want the data to be always replicated among the nodes. For example when moving to the new machines there's no data present on them and they need to get this data from the old ones. For that reason you need to wait with terminating the old brokers until they replicated all their data to the new ones. I'm wondering how did you solve that issue? – Domen P Jun 12 '15 at 09:00
  • I didn't actively tackle that. I think you need to handle this by the topic's replication factor: http://kafka.apache.org/documentation.html#basic_ops_add_topic – Tobi Jun 12 '15 at 09:06
  • To be honest, I'm not a very good expert concerning Kafka. I built the Docker image for testing purposes, and so far I didn't encounter problems. Before I'd move to production with that, I'd surely recommend to do furhter testing. – Tobi Jun 12 '15 at 09:14