1

I'm tring to use logstash-7.6.2 to sync my mongo with elasticsearch. I'm using dbschema jdbc driver.

input {
  jdbc{
    jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
    jdbc_driver_library => "/home/user/mongojdbc2.3.jar,/home/user/mongo-java-driver-3.12.6.jar,/home/user/gson-2.8.6.jar"
    jdbc_user => ""
    jdbc_password => ""
    jdbc_connection_string => "jdbc:mongodb://localhost:27027/test"
    statement => "db.mycollection.find()"
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9220"]
    manage_template => false
    index => "testing"
  }
  stdout { codec => rubydebug }
}

But I catch the next error:

Error: Java::JavaSql::SQLException: No suitable driver found for jdbc:mongodb://localhost:27027/test Exception: Sequel::DatabaseConnectionError Stack: java.sql.DriverManager.getConnection(java/sql/DriverManager.java:689) java.sql.DriverManager.getConnection(java/sql/DriverManager.java:247)

I also tried to use the mongo native java driver and the unity jdbc driver. I also tried to use different version of mongo, tried from localhost and from a remote server. I tried to use different versions of logstash. Everything comes down to this error.

Anton Magov
  • 186
  • 1
  • 14

1 Answers1

0

I've meet the same problems syncing MongoDB with Elastic and I want to share my experience with you. What I've tried:

  1. Longstash input MongoDB plugin. The easist way to integrate, needs minimum configuration. But it doesn't support "arrays of objects", it simply igrores this datastructure. You may flatten them on filter stage using ruby script (converting arr: [{a: 1, b: 2}, {a: 3, b: 4}] -> arr_1_a = 1, arr_1_b = 2, ... and so on).
  2. Logstash + jdcb mongo driver. The problem is that all mentions and instructions about this connection are a little bit outdated and all of them do not contain link to necessary Mongo Library. And libraries used in these instructions had been updated many times. Surely we have an option to search for older versions, but they do not support recent MongoDB versions. I've spent few days iterating through tens of different options (wisecoders dbschema, unity driver, many others). I've opened every library to investigate correct path to Java class, but none of them worked: every time I've meet different errors (and the most popular was No suitable driver found for jdbc:mongodb as yours).
  3. Monstache. It was my last hope and it worked flawlessly. The only thing I had to do is to convert my Mongo instance to ReplicaSet (it can be single-node for dev puproses) and prepare simple .toml config for monstache. Also it has ready-to-go Docker image. So I can personally recommend to use monstache for Mongo + Elastic syncing.

Hope this helps you.

akdev
  • 586
  • 2
  • 6
  • 18