0

I currently using Telegraf "eventhub_consumer" input plug-in to read events from EventHub that has 12 partitions. If I have generic Telegrah conf file where if i don't specify partition ids with one POD, it processes hardly 500 messages for each poll. I want to increase the throughput to 150k per minute. So, I thought of splitting the partition ids so that i can create multiple namespaces and process them in parallel. Even with that, i am consuming more resources.

[[inputs.eventhub_consumer]]
  data_format = "value"
  partition_ids = ["0","1","2","3"]
  data_type = "string"
  persistence_dir = "/var/lib/eventhub_persistence"
  [inputs.eventhub_consumer.tags]
    source="events"
[[inputs.eventhub_consumer]]
  data_format = "value"
  partition_ids = ["4","5","6","7"]
  data_type = "string"
  persistence_dir = "/var/lib/eventhub_persistence"
  [inputs.eventhub_consumer.tags]
    source="events"
[[inputs.eventhub_consumer]]
  data_format = "value"
  partition_ids = ["8","9","10","11"]
  data_type = "string"
  persistence_dir = "/var/lib/eventhub_persistence"
  [inputs.eventhub_consumer.tags]
    source="events"

So i decided to have above in telegraf.conf where in single Telegraf conf file i have 3 plug-in and created 3 replicas thinking each replica POD will process one plug-in each. With the below configuration, though i see 4 PODS running, I don't see even a single message being pulled. So wondering what's wrong in my setup. I deployed this via AKS deployment.

KNDM
  • 47
  • 7

0 Answers0