I'm new to Kafka, and am trying to build a service to service messaging platofrm on it. Here's my setup:
Kafka 0.9.0.1
Zookeeper 3.4.8
kafka-python 1.3.3
My application creates a KafkaProducer from which I send a stream of messages to a single topic with 6 partitions. I also create 7 KafkaConsumers (under a single group_id, 6 of which get assigned to the 6 partitions and one is left in an idle state (which is expected). While the producer is streaming, I increase the partition count to 7, with an expectation that the stream would not be distributed across 7 partitions and would wake up the idle consumer. However, it seems like the producer doesn't pick up the newly added partition until I re-initialize it by rebooting the application. I scale the partition count by running this:
kafka-topics --alter --zookeeper localhost:2181 --topic test --partitions 7
Is there a way for the producer to pick up the change in partition count without re-initializing it?
Here're the associated code snippets:
Producer
class Producer(threading.Thread):
daemon = True
def __init__(self, name, manager):
super(Producer, self).__init__()
self.producer = KafkaProducer(bootstrap_servers='localhost:9092')
def run(self):
while not self.killed:
if not self.q.empty():
self._busy()
self.producer.send('test', value=self.q.get())
else:
self._free()
Consumers
class Consumer(threading.Thread):
daemon = True
def __init__(self, name, manager):
super(Consumer, self).__init__()
self.consumer = KafkaConsumer(bootstrap_servers='localhost:9092',
group_id='test_group',
client_id="Consumer " + self.name)
self.consumer.subscribe(['test'])
def run(self):
while not self.killed:
messages = self.consumer.poll()
for topic, records in messages.iteritems():
print self.consumer.config['client_id'] + ": " + str(records)
