I want to be able to programmatically create a topic in Kafka using pykafka. I know that accessing the TopicDict will automatically create a topic if one does not exist but I don't know how to control the number of partitions/replicas with that. Also, it has a nasty bug where it ends up in an infinite loop if Kafka goes down. Basically I want to do something like the following:
create_topic('mytopic', partitions=2, replicas=3)
Pykafka is a Python implementation of Kafka producer and consumer API, what you want to achieve is an operation that in Kafka is performed using another API, the administration/operations API (actually a group of Java classes). I don't think Pykafka has an API/wrapper for that. What you may be observing is a topic being automatically created by Kafka. What you can do is to configure using properties the default number of partitions and replicas for automatically created topics.
You can do it with subprocess
If you install Kafka binaries, you can do something like this
from pykafka import KafkaClient
import subprocess
client = KafkaClient(hosts="localhost:9092")
subprocess.Popen("PATH/TO/KAFKA/BINARY/kafka_2.11-1.0.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic testtopic --replication-factor 1 --partitions 10".split())
