I currently using Telegraf "eventhub_consumer" input plug-in to read events from EventHub that has 12 partitions. If I have generic Telegrah conf file where if i don't specify partition ids with one POD, it processes hardly 500 messages for each poll. I want to increase the throughput to 150k per minute. So, I thought of splitting the partition ids so that i can create multiple namespaces and process them in parallel. Even with that, i am consuming more resources.
[[inputs.eventhub_consumer]]
data_format = "value"
partition_ids = ["0","1","2","3"]
data_type = "string"
persistence_dir = "/var/lib/eventhub_persistence"
[inputs.eventhub_consumer.tags]
source="events"
[[inputs.eventhub_consumer]]
data_format = "value"
partition_ids = ["4","5","6","7"]
data_type = "string"
persistence_dir = "/var/lib/eventhub_persistence"
[inputs.eventhub_consumer.tags]
source="events"
[[inputs.eventhub_consumer]]
data_format = "value"
partition_ids = ["8","9","10","11"]
data_type = "string"
persistence_dir = "/var/lib/eventhub_persistence"
[inputs.eventhub_consumer.tags]
source="events"
So i decided to have above in telegraf.conf where in single Telegraf conf file i have 3 plug-in and created 3 replicas thinking each replica POD will process one plug-in each. With the below configuration, though i see 4 PODS running, I don't see even a single message being pulled. So wondering what's wrong in my setup. I deployed this via AKS deployment.