2

I use a 5.1.2 verisón of logstash, filebeat, elasticsearch... "ELK"

I try send logs from tomcat server (catalina.out and apps-java logs) but can´t because have problems of config of logstash multiline filter/codec.

I follow this instructions
https://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html


Logstash.conf is this:

input {
    beats {
    port => 9000
    }
}

filter {
  if [type] == "tomcat-pro" {
    codec => "multiline" {
      patterns_dir => "/opt/logstash/patterns"
      pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
      negate => true
      what => "previous"
    }
  }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "tomcat-pro"
    }   
}

Logstash receives files of filebeat.
Filebeat.yml

filebeat.prospectors:
- input_type: log
  document_type: tomcat-pro
  paths:
  - /opt/tomcat-test/logs/catalina.out

When i start the service the console show me this:

[2017-01-26T13:10:33,712][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n    beats {\n    port => 9000\n    }\n}\n\nfilter {\n  if [type] == \"tomcat-pro\" {\n    codec => \"multiline\" {\n      patterns_dir => \"/opt/logstash/patterns\"\n      pattern => \"(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})\"\n      negate => true\n      what => \"previous\"\n    }\n  }\n}\n\noutput {\n    elasticsearch {\n        hosts => [\"localhost:9200\"]\n        index => \"tomcat-pro\"\n    }   \n}\n", :reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

Summary:

fetched an invalid config
reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

I read in google that is recommended to use multiline in filebeat rather than in logstash, but i dont config very well...

Someone can help me? :(

PD: Im spanish, sorry for "google translate". Si puedes responder en español, sería mucho mejor ;)

guare
  • 23
  • 1
  • 4

2 Answers2

3

I think doing the multiline processing in Filebeat is the way to go, so instead of debugging the Logstash configuration error you posted I will show a Filebeat configuration where Filebeat combines the lines before shipping the event.

If you were only using Logstash for the multiline filter, then you could just output directly to Elasticsearch from Filebeat. But if you do need to output to Logstash please follow the instructions for configuring Filebeat to be used with Logstash.

The pattern I use below hasn't been thoroughly tested so please test it against the actual logs.

filebeat.prospectors:
- document_type: catalina-wine-mixer
  paths:
  - /opt/tomcat-test/logs/catalina.out
  multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})|([J|F|M|A|M|S|O|N|D][a-z]{2} [0-9]{1,2}, [0-9]{2})'
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ['http://localhost:9200']
A J
  • 2,508
  • 21
  • 26
  • yes and it works good but filebeat send all ocurences of multiline again and again... explain: Multiline 1 Random text.... Multiline 2 Filebeat find Multiline1, send to elastic... Filebeat find Multiline2, send to elastic Multiline1 + Multiline2 – guare Jan 31 '17 at 17:19
  • Did you disable/remove the logstash output in the Filebeat config? Are you sending direct to ES and only ES? – A J Jan 31 '17 at 17:46
  • Sorry for not answer. The pattern work fine and logs send OK. Now i must filter fields that filebeat send. – guare Feb 15 '17 at 15:15
0

The answered pattern didn't quite work for me, for catalina/tomcat logs, i'm currently using the following filebeat.yml pattern:

multiline.pattern: '^[[:alpha:]]{3} [0-9]{2}, [0-9]{4}'
multiline.negate: true
multiline.match: after
olive_tree
  • 1,417
  • 16
  • 23