2

We are setting up elasticsearch, kibana, logstash and filebeat on a server to analyse log files from many applications. Due to reasons* each application log file ends up in a separate directory on the ELK server. We have about 20 log files.

  1. As I understand we can run a logstash pipeline config file for each application log file. That will be one logstash instance running with 20 pipelines in parallel and each pipeline will need its own beat port. Please confirm that this is correct?
  2. Can we have one filebeat instance running or do we need one for each pipeline/logfile?
  3. Is this architecture ok or do you see any major down sides?

Thank you!

*There are different vendors responsible for different applications and they run a cross many different OS and many of them will not or can't install anything like filebeats.

user1329339
  • 1,295
  • 1
  • 11
  • 26
  • filebeat should be at the same server that have application logs, and you can have one filebeat configured to read multiple files – ElasticCode Mar 26 '19 at 12:48
  • Thanks for your response but in my case it is not possible to install filebeat on the servers running the application. Instead the files will be read from the ELK server. Could you please share a link with documentation regarding filebeats and many logs? – user1329339 Mar 26 '19 at 13:15
  • check my answer – ElasticCode Mar 26 '19 at 13:31

1 Answers1

0

We do not recommend reading log files from network volumes. Whenever possible, install Filebeat on the host machine and send the log files directly from there. Reading files from network volumes (especially on Windows) can have unexpected side effects. For example, changed file identifiers may result in Filebeat reading a log file from scratch again.

Reference

We always recommend installing Filebeat on the remote servers. Using shared folders is not supported. The typical setup is that you have a Logstash + Elasticsearch + Kibana in a central place (one or multiple servers) and Filebeat installed on the remote machines from where you are collecting data.

Reference

For one filebeat instance running you can apply different configuration settings to different files by defining multiple input sections as below example, check here for more

filebeat.inputs:

- type: log

  enabled: true
  paths:
    - 'C:\App01_Logs\log.txt'
  tags: ["App01"]
  fields: 
    app_name: App01

- type: log
  enabled: true
  paths:
    - 'C:\App02_Logs\log.txt'
  tags: ["App02"]
  fields: 
    app_name: App02

- type: log
  enabled: true
  paths:
    - 'C:\App03_Logs\log.txt'
  tags: ["App03"]
  fields: 
    app_name: App03

And you can have one logstash pipeline with if statement in filter 

filter {

    if [fields][app_name] == "App01" {

      grok { }

    } else if [fields][app_name] == "App02" {

      grok { }

    } else {

      grok { }

    }
}

Condtion can be also if "App02" in [tags] or if [source]=="C:\App01_Logs\log.txt" as we send from filebeat

ElasticCode
  • 7,311
  • 2
  • 34
  • 45
  • Thanks for your feedback and links. 1. If I am not allowed to install filebeat on the target machine what alternatives do you suggest? 2. Since I am reading log files that look very different I was under the impression that I need to send them to different logstash pipelines for very different grok parsing and therefore it would require different filebeat.outputs but this is not allowed as far as I understand. Have I understood this wrong? – user1329339 Mar 26 '19 at 13:41
  • @user1329339 For point 1 I don't have suggestion right now, For point 2 for each input section you can have for example `tags: ["App01"]` and in logstash use if statment to check tags and do different grok parsing – ElasticCode Mar 26 '19 at 13:58
  • @user1329339 I have updated my answer for more details also – ElasticCode Mar 26 '19 at 14:21
  • Thanks again for your feedback. Unfortunately I can't install filebeats on different servers. Is an alternative to use the logstash file as input instead of beats and have different paths for different pipelines? – user1329339 Mar 26 '19 at 14:40
  • @user1329339 check this link for logstash file https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_reading_from_remote_network_volumes, Also normally how you will access current different servers log? – ElasticCode Mar 26 '19 at 14:47
  • The idea is to have one central server(linux) that mounts(NFS) a SAN directory as read only for each log file. Then filebeats will read each file and send it to logstash and then after processing it will be sent to elastic. – user1329339 Mar 26 '19 at 14:54
  • @user1329339 I think you should give it a try with NFS as purpose of NFS is to access files systems over a network as if they were local, AlsoI would like to know why you are not allowed to install filebeat on the target machine? BTW it doesn't need installation it will run as service and configured for one time only. – ElasticCode Mar 27 '19 at 10:11
  • It's politics. The company I work for have outsourced development to different companies and they are allowed to refuse to have 'my software' on their machines. – user1329339 Mar 27 '19 at 11:06