3

I already have Logstash configured to directly pick up files (Gatling logs), process them using custom logic, and send to Elasticsearch. And it's working fine. I'm now trying to setup Filebeat to send the files to it instead.

The basics seem to work as I see the new entires ending up in Elasticsearch, but they look all wrong. My theory is that Logstash is configured to parse Gatling logs, but Filebeat doesn't send the logs directly, but some JSON or other format containing the metadata as well, and Logstash needs to be reconfigured to parse this instead.

But... how do I find out what Filebeat is sending? Is the format documented somewhere?

kaqqao
  • 12,984
  • 10
  • 64
  • 118
  • Does that answer your question https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html? – baudsp Nov 24 '16 at 15:25
  • @baudsp Almost. It explains the metadata part, but not where the actual message (file content) goes. Or maybe I'm looking at this all wrong :( – kaqqao Nov 24 '16 at 15:30
  • No idea where this might be. Perhaps you have a codec on your input that causes problems. – baudsp Nov 24 '16 at 16:08
  • Make sure you are using the `beats` input in your Logstash config and that you have no codec applied to that input. The log line is sent the `message` field. Posting your Filebeat and Logstash config would help. – A J Nov 25 '16 at 19:49

1 Answers1

3

add the setting to filebeat.yml

you could see the output log as file in your path to see what happened.

output.file:
# Boolean flag to enable or disable the output module.
enabled: true

# Path to the directory where to save the generated files. The option is
# mandatory.
path: "your path"

# Name of the generated files. The default is `filebeat` and it generates
# files: `filebeat`, `filebeat.1`, `filebeat.2`, etc.
filename: filebeat

# Maximum size in kilobytes of each file. When this size is reached, and on
# every filebeat restart, the files are rotated. The default value is 10240
# kB.
#rotate_every_kb: 10000

# Maximum number of files under path. When this number of files is reached,
# the oldest file is deleted and the rest are shifted from last to first. The
# default is 7 files.
number_of_files: 7

if you want to see the network setting

please add the setting to see the log.

logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/mybeat
  name: mybeat.log
  keepfiles: 7
Joseph M Tsai
  • 568
  • 3
  • 12