filebeat send messages to certain index

Question

I have an installed pair elasticsearch - logstash - kibana, 2 clients: ELKclient1 and ELKclient2. Filebeat is installed on clients. I need that both clients write logs in separate index, ELKclient1 in index test-%{+YYYY.MM.dd, ELKclient2 in index test2-%{+YYYY.MM.dd (sending nginx access logs). For some reason logs from clients are written in both indexes, eg, from client ELKclient2 logs are written in both indexes test-%{+YYYY.MM.dd and test2-%{+YYYY.MM.dd (attachment 1 and attachement 2). Do you have any clue why its happening?

#config filebeat on client2
filebeat.inputs:
- type: log
  enabled: true
  paths:
      - /var/log/nginx/access.log
  fields:
    type: nginx_access
  fields_under_root: true
  scan_frequency: 5s

registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["ip-address_logstash:5044"]
    index: "test2-%{+YYYY.MM.dd}"
    bulk_max_size: 1024

shipper:
logging:
  to_syslog: false
  to_files: true
  level: info
  files:
    path: /var/log/filebeat
    name: filebeat.log

#config logstash output
output {
        elasticsearch {
            hosts    => "localhost:9200"
            index    => "test-%{+YYYY.MM.dd}"
        }
        #stdout { codec => rubydebug }
        elasticsearch {
            hosts    => "localhost:9200"
            index    => "test2-%{+YYYY.MM.dd}"
        }
        #stdout { codec => rubydebug }

}

Answer 1

In order to make both clients write logs in a separate index, Take the workflow idea in the below picture, You need to add a tag to differentiate the logs coming from different servers.

Considering your requirement in your question one of the ways is to put the following code in the output section of your logstash config file.

output {
  if [beat][hostname] == "ELKclient1"
    elasticsearch {
            hosts    => "localhost:9200"
            index    => "test-%{+YYYY.MM.dd}"
        }
  else if [beat][hostname] == "ELKclient2"
    elasticsearch {
            hosts    => "localhost:9200"
            index    => "test2-%{+YYYY.MM.dd}"
        }
  else 
    stdout { 
      codec => rubydebug 
    }
}