1

In my Logstash I have one pipeline that runs 2 SQL queries to download data. Below conf file for the pipeline:

input {

  jdbc {
        jdbc_driver_library => "/opt/logstash/lib/ojdbc8.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        #Hidden db connection details
        statement_filepath => "/etc/logstash/queries/transactions_all.sql"
        type => "transactions"
  }
  jdbc {
        jdbc_driver_library => "/opt/logstash/lib/ojdbc8.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        #Hidden db connection details
        statement_filepath => "/etc/logstash/queries/snow_db_stats_ts_all.sql"
        type => "db_stats_ts"
  }

output{
       if [type] == "transactions" {
         elasticsearch {
             index => "servicenow_oraaspoc-%{+YYYY.MM.dd}"
             hosts => ["localhost:9200"]
         }
       }
       if [type] == "db_stats_ts" {
        elasticsearch {
            index => "snow_db_stats_ts-%{+YYYY.MM.dd}"
            hosts => ["localhost:9200"]
        }
       }
  stdout {     
          codec => rubydebug
  }

}

I can see in the console that everything works fine except index with type transactions is never saved to Elasticsearch. This condition if [type] == "transactions" { is never true and the second condition works without any problems. I tried to run pipeline just with transactions index without if condition and it worked fine. For some reason this one if condition is not working but why?

I have found one ridiculous workaround but it won't work if I encounter another index with this problem:

   if [type] == "db_stats_ts" { .. } else {
       elasticsearch {
          index => "servicenow_oraaspoc-%{+YYYY.MM.dd}"
          hosts => ["localhost:9200"]
       }
   }
Michael Dz
  • 3,655
  • 8
  • 40
  • 74

1 Answers1

3

ike Thomas Decaux mentioned you can use tags instead of type, btw you can use as many tags as you want for each jdbc block. Remove types from config file with tags like in the example below:

tags => ["transactions"]

tags => ["db_stats_ts"]

Your output block should look like that:

output{
       if "transactions" in [tags] {
         elasticsearch {
           index => "servicenow_oraaspoc-%{+YYYY.MM.dd}"
           hosts => ["localhost:9200"]
         }
       }
       if "db_stats_ts" in [tags] {
        elasticsearch {
          index => "snow_db_stats_ts-%{+YYYY.MM.dd}"
          hosts => ["localhost:9200"]
        }
       }
}
Dioxis
  • 46
  • 3