Connect to multiple databases dynamically using Logstash JDBC Input plugin

Question

I am using Logstash JDBC input plugin to read data from database and index it into Elastic Search. I have separate database for each customer and I want to connect to them one by one dynamically to fetch data?

Is there any provision or parameter in JDBC-Input Plugin or Logstash to connect to multiple databases?

e.g

input {
  jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/MYDB"
    //MYDB will be set dynamically. 
    jdbc_user => "mysql"
    parameters => { "favorite_artist" => "Beethoven" }
    schedule => "* * * * *"
    statement => "SELECT * from songs where artist = :favorite_artist"
  }
}

Only solution I can think of is writing script that will update logstash config to connect to specified databases one by one and run logstash through it.

maybe the easiest way would be to have multiple jdbc input. How many customers do you have ? — maximede, Mar 29 '16 at 16:06
Approx 500, Customer database server details are stored in one central database. — ni3ns, Mar 29 '16 at 16:47
@ni3ns did you find a solution for it? I have the same problem statement. — Sankalp Tambe, Jun 11 '20 at 06:23

score -1 · Answer 1 · answered Mar 26 '18 at 09:01

Let me update this - for the same kind of purpose, I used two input JDBC sections, but only first section considered.

input {
    jdbc {
        jdbc_connection_string => "XXXX"
        jdbc_user => "XXXX"
        jdbc_password => "XXXX"
        statement => "select * from product"
        jdbc_driver_library => "/usr/share/logstash/ojdbc7.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        }
 jdbc {
        jdbc_connection_string => "YYYY"
        jdbc_user => "YYYYY"
        jdbc_password => "YYYY"
        statement => "select * from product"
        jdbc_driver_library => "/usr/share/logstash/ojdbc7.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        }
}
output {
    elasticsearch {
        hosts => "localhost:9200"
        user => "XXX"
        password => "XXXX"
        index => "XXXX"
        document_type => "XXXX"
        }
}

--

Connect to multiple databases dynamically using Logstash JDBC Input plugin

1 Answers1