Logstash jdbc not sending data

Question

I'm trying to export datas from a mysql table to elastic search with logstash and the jdbc mysql driver with every process in a docker container. My problem is (with no error) their is nothing sent to elastic search.

My Dockerfile :

FROM elastic/logstash:6.3.0

ENV https_proxy=
ENV http_proxy=

COPY ./mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar /tmp/mysql-connector-java-5.1.46.jar
COPY ./logstash.conf /tmp/logstash.conf
COPY ./logstash.yml /usr/share/logstash/config/logstash.yml

RUN logstash-plugin install logstash-input-jdbc

I run it with this command :

docker run -d --rm --name=logstach -v /data/logstash:/home/logstash logstash bin/logstash -f /tmp/logstash.conf

And here is my logstash.conf :

input {
        jdbc {
            jdbc_driver_library => "/tmp/mysql-connector-java-5.1.46.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_connection_string => "jdbc:mysql://0.0.2.22:3306/itop_db"
           jdbc_user => "admin"
            jdbc_password => "password"
            statement => "SELECT * FROM contact”
        }
    }
    output {
         elasticsearch {
            index => "contact"
            document_type => "data"
            document_id => "%{id}"
            hosts => "127.0.0.1:9200"
        }
        stdout { codec => json_lines }
    }

Every thing seems to accomplish well, except their is no new index in elastic search http://localhost:9200/_cat/indices?v

This is the output I have when i run logstash :

logstash execution output

logstash error 2

Are you sur that the address 0.0.2.22:3306 is recheable from the logstash container? Try to execut a `ping`, `telnet` or `curl` command — andolsi zied, Aug 06 '18 at 12:18
The mysql IP is good, I just changed the 2 first numbers by 0.0. I can connect to mysql with the infos in the jdbc input from the same machine I execute the docker run. I mean this works : `mysql -h "0.0.2.22" -u admin -ppassword ` — Kalus, Aug 06 '18 at 12:22
Yes, done. this is the output of the command I use to run logstash — Kalus, Aug 06 '18 at 12:38

score 1 · Answer 1 · answered Aug 06 '18 at 12:45

1

"SELECT * FROM contact” <-- this could be the problem. I imagine you copied this from the internet? Change ” to "

answered Aug 06 '18 at 12:45

dwjv

1,227
9
15

Oh, thank you. Did not seen that. The sql command was copied from a word document I think. I now have connection refused from elastic search, but I'll try to resolv it on my own – Kalus Aug 06 '18 at 12:52
I think the default docker image of ES has x-pack enabled, might need to disable it or provide creds. – dwjv Aug 06 '18 at 12:56
You think it could be my problem ? I updated my first message – Kalus Aug 06 '18 at 13:01
Ah, I think you have the IP of ES wrong? Unless it's running on the same node as Logstash (doesn't appear to be from the Dockerfile). – dwjv Aug 06 '18 at 13:06
Yes, I am running thoses services on the same machine. I run ES with the default image with no modification though. – Kalus Aug 06 '18 at 13:10
But ES is running in a different container (as I can't see any mention of it in your Logstash Dockerfile)? If so, `127.0.0.1` is not the right IP to connect to. You need the IP of the ES container. – dwjv Aug 06 '18 at 13:12
I think as the container is listening on 0.0.0.0:9200, I can give its direct ip (172.17.0.2 in my case) or the 127.0.0.1 and both should work. – Kalus Aug 06 '18 at 13:16
1

It is now working, I have made some errors while testing and it is fixed. I have a new index (contact) in my ES instance. Thanks you – Kalus Aug 06 '18 at 13:18

score 0 · Answer 2 · answered Aug 06 '18 at 13:26

0

In addition to the error in my SQL statement, I needed to specify the host ip of the container (172.17.0.2 in my case) instead of using 127.0.0.1:9200

answered Aug 06 '18 at 13:26

Kalus

1
2

Logstash jdbc not sending data

2 Answers2