Logstash - Data from Kafka to ES

Question

Using logstash 5.0.0, Taking kafka source as the input -> taking the data and producing the output in Elasticsearch. (ElasticSearch version 5.0.0)

Logstash conf:

input{
kafka{
bootstrap_servers => "XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092"
topics => ["a-data","f-data","n-data"]
group_id => "sound"
auto_offset_reset => "earliest"
consumer_threads => 2
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
hosts => [ "XXX.XXX.XX.XXX:9200" ]
}
}

When I run the below configuration , i am getting this following error.

$ ./logstash -f sound.conf
Sending Logstash logs to /logstash-5.0.0/logs which is now configured vi   a log4j2.properties.
[2017-01-17T10:53:29,273][ERROR][logstash.agent           ] fetched an invalid c   onfig {:config=>"input{\nkafka{\nbootstrap_servers => \"XX.XXX.XXX.XX:9092\",\"XXX.XXX.XX.XXX:9092\",\"XXX.XXX.XX.XXX:9092\"\ntopics => [\"a-data\",\"f-data\   ",\"n-data\"]\ngroup_id => \"sound\"\nauto_offset_reset => \"earliest\"\nc   onsumer_threads => 2\n}\n}\nfilter{\njson{\nsource => \"message\"\n}\n}\noutput    {\nelasticsearch {\nhosts => [ \"XX.XX.XXX.XX:9200\" ]\n}\n}\n\n", :reason=>"Ex   pected one of #, {, } at line 3, column 40 (byte 54) after input{\nkafka{\nboots   trap_servers => \"XX.XX.XXX.XX:9092\""}

Can anyone help me with this configuration.

score 0 · Answer 1 · answered Jan 17 '17 at 12:26

0

Shouldn't your topic be topics which is an array, where you've inserted the values as a hash:

topics => ["a-data","f-data","n-data"] <-- try changing this line

answered Jan 17 '17 at 12:26

Kulasangar

9,046
5
51
82

In your `kafka` input I don't see the last three synopsis being included in the 5.0 [version](https://www.elastic.co/guide/en/logstash/5.0/plugins-outputs-kafka.html#_synopsis_75). – Kulasangar Jan 17 '17 at 12:35
Darth_Vader - I dont get you. last three synopsis? – Soundarya Thiagarajan Jan 17 '17 at 12:40
I mean these three: group_id => "sound" auto_offset_reset => "earliest" consumer_threads => 2 – Kulasangar Jan 17 '17 at 16:55
I excluded those three lines and tried as well. didnt work – Soundarya Thiagarajan Jan 18 '17 at 03:45
can you help me with this. I tried removing group_id => "sound" auto_offset_reset => "earliest" consumer_threads => 2 .. but doesnt seem to work. can anyone help ? – Soundarya Thiagarajan Jan 18 '17 at 05:31

Logstash - Data from Kafka to ES

1 Answers1