2

We have standard log lines in our Spring Boot web applications (non json). We need to centralize our logging and ship them to an elastic search as json.

(I've heard the later versions can do some transformation)

Can Filebeat read the log lines and wrap them as a json ? i guess it could append some meta data aswell. no need to parse the log line.

expected output : {timestamp : "", beat: "", message: "the log line..."}

i have no code to show unfortunately.

oygen
  • 467
  • 1
  • 7
  • 13

2 Answers2

1

filebeat supports several outputs including Elastic Search.

Config file filebeat.yml can look like this:

# filebeat options: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/../file.err.log

processors:
   - drop_fields:
      # Prevent fail of Logstash (https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-6.3.html#custom-template-non-versioned-indices)
      fields: ["host"]
   - dissect:
      # tokenizer syntax: https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html.
      tokenizer: "%{} %{} [%{}] {%{}} <%{level}> %{message}"
      field: "message"
      target_prefix: "spring boot"

fields:
  log_type: spring_boot

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD"
vladimir
  • 13,428
  • 2
  • 44
  • 70
  • Thank you for your answer! we may be sending it first to kafka, will the config be almost the same then? – oygen Jun 12 '19 at 07:30
  • Kafka-output is supported too - just look at docs (https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html) – vladimir Jun 12 '19 at 07:34
  • thanks, i checked the docs but the problem is the json transformation before kafka/elastic. they will both expect json. – oygen Jun 12 '19 at 07:37
  • It looks like the output format is defined in [codec settings](https://www.elastic.co/guide/en/beats/filebeat/master/configuration-output-codec.html); as I understood for kafka-output used [json-codec by default](https://www.elastic.co/guide/en/beats/filebeat/6.7/kafka-output.html#_literal_codec_literal). Could you check it and notify about result of testing? – vladimir Jun 12 '19 at 08:11
0

Well it seems to do it by default. this is my result when i tried it locally to read log lines. it wraps it exactly like i wanted.

{  
   "@timestamp":"2019-06-12T11:11:49.094Z",
   "@metadata":{  
      "beat":"filebeat",
      "type":"doc",
      "version":"6.2.4"
   },
   "message":"the log line...",
   "source":"/Users/myusername/tmp/hej.log",
   "offset":721,
   "prospector":{  
      "type":"log"
   },
   "beat":{  
      "name":"my-macbook.local",
      "hostname":"my-macbook.local",
      "version":"6.2.4"
   }
}
oygen
  • 467
  • 1
  • 7
  • 13