3

I am trying to load data from filebeat into logstash. While loading , while running the command-> 

bin/logstash -f first-pipeline.conf --config.reload.automatic

, following error is encountered:

[2018-06-05T11:30:43,987][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-06-05T11:30:44,047][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x969dfe run>"}
[2018-06-05T11:30:44,083][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-06-05T11:30:44,112][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-06-05T11:32:05,045][INFO ][org.logstash.beats.BeatsHandler] [local: 0:0:0:0:0:0:0:1:5044, remote: 0:0:0:0:0:0:0:1:31903] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69

first-pipeline.conf file is: 

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
stdout { codec => rubydebug }
}

Filebeat.yml file:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - \C:\PATH-TO-DOC\elasticDoc\logstash-tutorial-dataset.log
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Sample dataset of logstash-tutorial-dataset.log :

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

What is the cause of this error? This question has already been asked before but there were no replies. Please also let me know where i could polish my concepts in logstash and filebeat more. I am a beginner.

Umang Bhargava
  • 43
  • 2
  • 6
  • Other than the logstash output, what else is happening? Does the logstash process stops? The messages are not processed anymore? – baudsp Jun 05 '18 at 10:10
  • When i Tried to send data from stdin to elasticsearch from logstash then it is working fine. But data cannot be send from filebeat to logstash. I donot think the messages are being processes at all. – Umang Bhargava Jun 07 '18 at 06:07
  • Logstasah was showing this at last : [2018-06-06T13:22:38,140][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84 – Umang Bhargava Jun 07 '18 at 06:18
  • Perhaps it's a compatibility issue? ie the version of logstash you are using does not support the version of filebeat you are using – baudsp Jun 07 '18 at 07:37
  • 1
    I have latest logstash and filebeat versions i.e. 6.2.4 – Umang Bhargava Jun 07 '18 at 07:51

1 Answers1

1

The problem was with my filename in filebeat.yml . The extension was not needed. Also in first-pipeline.conf file, i removed codec and send my logs directly to elastic search and it started working for me.

Umang Bhargava
  • 43
  • 2
  • 6