Logstash multiple inputs multiple outputs

Question

I'm trying to sync data between MySQL and Elasticsearch with Logstash.

I set multiple jdbc inputs and multiple outputs to different elasticsearch indexes ... and something I am doing wrong because everything is going to the else block.

Here is my config:

 input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql:127.0.0.1:3306/whatever"
        jdbc_user => "xxx"
        jdbc_password => "yyy"
        jdbc_driver_library => "mysql-connector-java-5.1.41.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        schedule => "* * * * *"
        statement => "SELECT * from table1 WHERE updated_at > :sql_last_value order by updated_at"
        use_column_value => true
        tracking_column => updated_at
        type => "table1"
        last_run_metadata_path => "/opt/logstash-5.4.0/sql-last-values/table1"
    }
      jdbc {
        jdbc_connection_string => "jdbc:mysql:127.0.0.1:3306/whatever"
        jdbc_user => "xxx"
        jdbc_password => "yyy"
        jdbc_driver_library => "mysql-connector-java-5.1.41.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        schedule => "* * * * *"
        statement => "SELECT * from table2 WHERE updated_at > :sql_last_value order by updated_at"
        use_column_value => true
        tracking_column => updated_at
        type => "table2"
        last_run_metadata_path => "/opt/logstash-5.4.0/sql-last-values/table2"
    }
}
output {
    if [type] == "table1" {
           elasticsearch {
                hosts => ["localhost:9200"]
                index => "table1"
                document_type => "table1"
                document_id => "%{id}"
        }
        file {
                codec => json_lines
                path => "/opt/logstash-5.4.0/logs/table1.log"
        }

    } else if [type] == "table2" {
           elasticsearch {
                hosts => ["localhost:9200"]
                index => "table2"
                document_type => "table2"
                document_id => "%{id}"
        }
    } else {
         file {
                codec => json_lines
                path => "/opt/logstash-5.4.0/logs/unknown.log"
            }

    }
}

What am I doing wrong ? everything is going to the else block, to the /opt/logstash-5.4.0/logs/unknown.log

Is wrong my approach ? Should I have multiple files ?

thank you in advance

score 25 · Accepted Answer · edited Nov 22 '18 at 10:53

25

Solution found!

I used tags instead of type

input {
jdbc { 
...
tags => "table1"
...
}
jdbc { 
...
tags => "table2"
...
}
}
output {
  if "table1" in [tags] {

}

https://discuss.elastic.co/t/solved-multiple-logstash-config-file/51692/10

edited Nov 22 '18 at 10:53

ScottSummers

310
1
13

answered May 31 '17 at 20:38

JPG

750
2
8
23

I was having a similar issue and using tags fixed it. It appears that "type" can be used in a log field and will override the input's value e.g. with winlogbeat where "type" could be the even type for the log. – Timothy Gonzalez Sep 08 '17 at 14:05
You can also consider adding metadata and removing the tags if you don't want to have a tags field in the documents that are output. For example, you could add metadata corresponding to each tag, then remove the tags, and then use these metadata fields to drive different documents to different outputs. I have written a blog post that demonstrates this approach at https://alexmarquardt.com/2018/08/31/using-logstash-to-drive-filtered-data-from-a-single-source-into-multiple-output-destinations/. Disclaimer: I am a Consulting Engineer at Elastic. – Alexander Marquardt Sep 02 '18 at 12:54

Logstash multiple inputs multiple outputs

1 Answers1