0

I have data coming to telegraf and it has multiple fields. I want to be able to have one inputs.exec plugin and multiple outputs.kafka plugin to send each field to respective kafka topics.

My data is in this format:

{
    "field1": [
        {
            "abc" : 0,
            "efg" : 1,
            "hij" : 4,
            "jkl" : 5
        }
    ],
    "field2": [
        {
            host : "admin1",
            timestamp: 1682314679774
        },
        {
            host : "admin2",
            timestamp: 1682314679775
        },
        {
            host : "admin3",
            timestamp: 1682314679773
        }
    ]
}

I want to send field1 to "field1" kafka topic and field2 to "field2" kafka topic. My expected output is:

In field1 kafka topic: { "abc" : 0, "efg" : 1, "hij" : 4, "jkl" : 5 }

In field2 kafka topic: { host : "admin1", timestamp: 1682314679774 }, { host : "admin2", timestamp: 1682314679775 }, { host : "admin3", timestamp: 1682314679773 }

How do i implement this in telegraf configuration file?

I tried writing the telegraf conf file in this format. But it is not working.

[[outputs.kafka]]
    brokers = ["admin:9091"]
topic = "field1"
data_format = "json"
flush_interval = "1m"

[[outputs.kafka]]
    brokers = ["admin:9092"]
## Kafka topic for producer messages
topic = "field2"
data_format = "json"
flush_interval = "1m"

[[inputs.exec]]
interval = "1m"
commands = ["/opt/clustertest/bin/script.py"]
timeout = "10s" # I want the script to execute every ten seconds
data_format = "json"
flush_interval = "1m"
json_query = "{'field1': [], 'field2': []}"

Can anyone please help?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Groot
  • 1
  • I don't know if this is possible. 1) You have no filter to separate your data, so if this did work, all data would go to both topics 2) you can use tools like MirrorMaker to replicate data from one Kafka cluster to another 3) You seem to be using the same `admin` hostname, but two ports... Why are you running two kafka servers on the same host? – OneCricketeer Apr 25 '23 at 19:18
  • Okay, how do I filter the data? I have added the json_query to filter but it is not doing that. I want separate fields to go to separate topics, what is the point of replicating? Sorry, two kafka servers was a mistake. It is one server. – Groot May 25 '23 at 06:57
  • If it's one server, please [edit] to correct your config and how you've tried to add telegraf filter, rather than a command execution input. Also, I don't think you can define `outputs.kafka` more than once anyway... Besides, why exactly do you even need telegraf? There are Python Kafka clients where you can just run the Python script on a 10s schedule (which is interval setting, not timeout), writing to as many topics as you want – OneCricketeer May 25 '23 at 12:15

0 Answers0