2

Trying to sync between MongoDB and Elasticsearch with MongoDbJdbcDriver by following this answer. I use docker-compose for development, and this is how it looks like for logstash:

  logstash:
    image: logstash:7.9.1
    container_name: logstash
    volumes:
      - ./logstash/jars/gson-2.8.6.jar:/usr/share/logstash/logstash-core/lib/jars/gson-2.8.6.jar:ro
      - ./logstash/jars/mongojdbc2.3.jar:/usr/share/logstash/logstash-core/lib/jars/mongojdbc2.3.jar:ro
      - ./logstash/jars/mongo-java-driver-3.12.6.jar:/usr/share/logstash/logstash-core/lib/jars/mongo-java-driver-3.12.6.jar:ro
      - ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
      - ./logstash/pipeline/mongo-to-elasticsearch.conf:/usr/share/logstash/pipeline/mongo-to-elasticsearch.conf
    command: logstash
    depends_on:
      - elasticsearch

I run docker-compose up and it gives this error:

...
logstash         |   Error: unable to load mongojdbc2.3.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
logstash         |   Exception: LogStash::PluginLoadingError
...

I've checked file permission of mongojdbc2.3.jar on my machine, given read and write for me and docker group. However when I check inside logstash container, owner is not root but logstash:

// ls -l /usr/share/logstash/logstash-core/lib/jars

...
-rw-r--r-- 1 logstash logstash  2315317 Jul 24 15:13 mongo-java-driver-3.12.6.jar
-rw-r--r-- 1 logstash logstash    83500 Sep  8 01:30 mongojdbc2.3.jar
-rw-rw-r-- 1 logstash root       107210 Sep  1 23:32 org.eclipse.core.commands-3.6.0.jar
...

I tried changing ownership inside container but I couldn't do it since I don't have the sudo permission.

Please help, how I can I sync between MongoDB and Elasticsearch successfully?

Stack: MongoDB (v4.4.1), Logstash (v7.9.1), Docker (v19.03.12), Docker-compose (v1.27.3)

Swix
  • 1,883
  • 7
  • 33
  • 50

1 Answers1

1

Try to create new image with jars in it

  logstash:
    build: ./logstash/
    container_name: logstash
    volumes:
      - ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
      - ./logstash/pipeline/mongo-to-elasticsearch.conf:/usr/share/logstash/pipeline/mongo-to-elasticsearch.conf
    command: logstash
    depends_on:
      - elasticsearch

./logstash/Dockerfile


FROM docker.elastic.co/logstash/logstash:7.9.1

COPY ./jars/ /usr/share/logstash/logstash-core/lib/jars
romanlv
  • 1,432
  • 15
  • 15