1

Trying to create multiple indexes for elasticsearch in logstash. But my "if conduction" is not creating any single index, without if conduction it is working fine.

But if I'm using input as file and in logstash without using filebeat then it is working as per my expectation. Can anyone help me for resolution. 

###filebeat.yml###
=============
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/vinit/pache/*.log
  fields:
    log_type: apache-log

- type: log
  enabled: true
  paths:
    - /home/user/vinit/boss/*.log
  fields:
    log_type: jboss-log
  fields_under_root: true

###pipeline-conf.conf###
==================

input {
  beats {
    port => 5044
  }
}

filter {
    grok {
           match => { "message" => "^%{IP:CLIENT_IP} (?:-|%{USER:IDEN}) (?:-|%{USER:AUTH}) \[%{HTTPDATE:CREATED_ON}\] \"(?:%{WORD:REQUEST_METHOD} (?:/|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})?|-)\" %{NUMBER:RESPONSE_CODE} (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
    add_field => {
                "LOG_TYPE" => "api-log"
        }
    overwrite => [ "message" ]
    }
    grok {
           match => { "message" => "%{HTTPDATE:CREATED_ON}%{NOTSPACE}%{SPACE} (?:-|%{IP:CLIENT_IP})%{SPACE} %{NOTSPACE}(?:-|%{WORD:REQUEST_METHOD}%{SPACE}) (?:-|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})%{NOTSPACE}(?:-|%{GREEDYDATA:OTHER_INFO}) (?:-|%{NUMBER:RESPONSE_CODE}) (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
    add_field => {
                "LOG_TYPE" => "web-log"
        }
    overwrite => [ "message" ]
    }
    grok {
           match => { "message" => "%{TIME:CREATED_ON}%{SPACE}\[(?<THREAD>[^\]]+)?\] %{WORD:METHOD}%{SPACE}%{JAVACLASS:CLASS} - (?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<extra>(.|\r?\n)+))?"}
    add_field => {
                "LOG_TYPE" => "jboss-log"
        }
    overwrite => [ "message" ]
    }
}
output {
   if [fields][log_type] == "apache-log"{
     elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "server-logs-apache"
     }
  }
   if [fields][log_type] == "jboss-log" {
     elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "server-logs-jboss"
     }
  }
   stdout { codec => rubydebug }    
}

##Also Tried##
==============
output {
       if "apache-log" in [fields][log_type] {
         elasticsearch {
         hosts => ["localhost:9200"]
         manage_template => false
         index => "server-logs-apache"
         }
      }
       if "jboss-log" in [fields][log_type] {
         elasticsearch {
         hosts => ["localhost:9200"]
         manage_template => false
         index => "server-logs-jboss"
         }
      }
       stdout { codec => rubydebug }    
    }

I'm expecting result as indexes : server-logs-apache, server-logs-jboss but actual output is empty.

Vinit Jordan
  • 313
  • 1
  • 5
  • 13

1 Answers1

0

You're adding field which is capital case

add_field => { "LOG_TYPE" => "web-log" }

and Elasticsearch separates field written with different case. You should add field lowercased - "log_type"

Mysterion
  • 9,050
  • 3
  • 30
  • 52
  • I changed it to custom_type and it's just a static value. Help to use filebeat variable in logstash. – Vinit Jordan Jan 13 '19 at 14:54
  • I had 3 conf files of logstash i.e first-input.conf, second-filter.conf, third-output.conf. I created one file and combine these three conf files configurations in beat-data.conf file. And now it's working as I want. – Vinit Jordan Jan 14 '19 at 16:41