I am encountering a problem with a SpringBoot web service failing to successfully authenticate to a bare metal MariaDB 10.3.8 database when the SpringBoot service is run from inside a Docker container. The same SpringBoot service JAR connects successfully to the DB when run outside on the command line. Here are the particulars:
- server is Fedora 30 running on 192.168.99.10
- MariaDB version 10.3.8 bound to 0.0.0.0 (not Dockerized)
- mysql-connector-5.1.41
- Java OpenJDK 13 running on Fedora 30
- Docker version 19.03.0-rc3
- Docker image openjdk:13-alpine
The SpringBoot app is coded to use environment variables to define the location and credentials of the MariaDB database instance.
The class within the Java service that establishes the JDBC pool has extra instrumentation to confirm the environment values are being set and are available at the point the JDBC pool is instantiated:
@Autowired
public void setDataSource(DataSource dataSource) {
// pull environment variables being used and log them for debugging
String dbhost = System.getenv("DOCKENV_MYSQL_HOST");
String dbport = System.getenv("DOCKENV_MYSQL_PORT");
String dbuser = System.getenv("DOCKENV_MYSQL_USERID");
String dbpw = System.getenv("DOCKENV_MYSQL_PASSWORD");
thisLog.info("DOCKENV_MYSQL_HOST=" + dbhost + " DOCKENV_MYSQL_PORT=" + dbport + " DOCKENV_MYSQL_USERID=" + dbuser
+ " DOCKENV_MYSQL_PASSWORD=" + dbpw);
jdbcTemplate = new JdbcTemplate(dataSource);
}
Here's what the SpringBoot service logs when executed and invoked with a curl command at the Linux prompt layer.
[mdh@fedora1 ~/gitwork/kuberdepends]$ printenv | grep DOCK
DOCKENV_MYSQL_PORT=3306
DOCKENV_MYSQL_PASSWORD=badpassword
DOCKENV_MYSQL_HOST=192.168.99.10
DOCKENV_MYSQL_USERID=dependsapp
[mdh@fedora1 ~/gitwork/kuberdepends]$ java -jar target/depends.jar 1>/dev/null 2>/dev/null&
[1] 7814
[mdh@fedora1 ~/gitwork/kuberdepends]$
[mdh@fedora1 ~/gitwork/kuberdepends]$ curl -H "Content-type: application/json" -X GET http://127.0.0.1:8080//depends/api/projects/34
{"project_id":34,"projectstatus_id":0,"clientbusunit_id":2,"clientbusdept_id":2,"factorybusunit_id":11,"factorybusdept_id":23,"projectname":"Ent Portal Unification","shortdescription":"Merge enterprise portals into spectrum.net","longdescription":"Longer desc here","hascapitalspend":"Y","hasexpensespend":"Y","capitalledger":"","expenseledger":"","clientpriority":1,"deliverypriority":1,"restricttodept":"N","restricttomembers":"N","createdatetime":"2018-05-20 20:51:22.0","updatedatetime":null}[mdh@fedora1 ~/gitwork/kuberdepends]$
[mdh@fedora1 ~/gitwork/kuberdepends]$
[mdh@fedora1 ~/gitwork/kuberdepends]$
[mdh@fedora1 ~/gitwork/kuberdepends]$ cat /logs/springboot/dependsLog.txt | grep dependsapp
2019-08-24 13:33:45.077 INFO 7814 --- [main] com.charter.depends.dao.ProjectsDAO : DOCKENV_MYSQL_HOST=192.168.99.10 DOCKENV_MYSQL_PORT=3306 DOCKENV_MYSQL_USERID=dependsapp DOCKENV_MYSQL_PASSWORD=badpassword
[mdh@fedora1 ~/gitwork/kuberdepends]$
That working version of the JAR is wrapped as a Docker container with the following Dockerfile:
[mdh@fedora1 ~/gitwork/kuberdepends]$ cat Dockerfile.openjdk13alpine
# Docker file -- use as micro-sized test image with basic Linux commands
# 1) use alpine image as starting slice
# 2) add packages: iputils, busybox-extras (for telnet), mariadb-client (testing MariaDB access)
# 3) starts the SpringBoot app in depends.jar as java -jar /opt/mdhlabs/depends.jar
FROM openjdk:13-alpine
RUN mkdir /opt/mdhlabs
COPY ./target/depends.jar /opt/mdhlabs/depends.jar
WORKDIR /opt/mdhlabs
RUN apk update && apk add iputils && apk add busybox-extras && apk add mariadb-client
CMD ["java", "-jar","/opt/mdhlabs/depends.jar"]
[mdh@fedora1 ~/gitwork/kuberdepends]$
[mdh@fedora1 ~/gitwork/kuberdepends]$ docker build -t kuberdepends-alp13 -f Dockerfile.openjdk13alpine .
(stuff omitted here for brevity...)
Successfully tagged kuberdepends-alp13:latest
[mdh@fedora1 ~/gitwork/kuberdepends]$
After stopping that Springboot process and running the Dockerized version port mapping the internal 8080 to an outer 7777 from the main Linux environment, here's what is logged.
[mdh@fedora1 ~/gitwork/kuberdepends]$ docker run --network=host -p 7777:8080 -d -e DOCKENV_MYSQL_HOST='192.168.99.10' -e DOCKENV_MYSQL_PORT='3306' -e DOCKENV_MYSQL_USERID='dependsapp' -e DOCKENV_MYSQL_PASSWORD='badpassword' --name kuberdepends-container kuberdepends-alp13
WARNING: Published ports are discarded when using host network mode
293d36ed488ca076d050f9579bb54979ff9e469a9d4429ad58204f069dbfd358
[mdh@fedora1 ~/gitwork/kuberdepends]$
[mdh@fedora1 ~/gitwork/kuberdepends]$ curl -H "Content-type: application/json" -X GET http://127.0.0.1:7777//depends/api/projects/34
curl: (7) Failed to connect to 127.0.0.1 port 7777: Connection refused
[mdh@fedora1 ~/gitwork/kuberdepends]$
If I access the Docker container, perform a wget to the internal 8080 listener port and inspect the log file generated by the SpringBoot service, here is the output related to the attempt to instantiate the JDBC pool.
[mdh@fedora1 ~/gitwork/kuberdepends]$ docker exec -it kuberdepends-container sh
/opt/mdhlabs # wget --header "Content-type: application/json" http://127.0.0.1:8080//depends/api/projects/34
Connecting to 127.0.0.1:8080 (127.0.0.1:8080)
wget: server returned error: HTTP/1.1 500
/opt/mdhlabs #
/opt/mdhlabs # cat /logs/springboot/dependsLog.txt | grep dependsapp
2019-08-24 18:52:08.956 INFO 1 --- [main] com.charter.depends.dao.ProjectsDAO : DOCKENV_MYSQL_HOST=192.168.99.10 DOCKENV_MYSQL_PORT=3306 DOCKENV_MYSQL_USERID=dependsapp DOCKENV_MYSQL_PASSWORD=badpassword
java.sql.SQLException: Access denied for user 'dependsapp'@'127.0.0.1' (using password: YES)
2019-08-24 18:56:14.796 ERROR 1 --- [http-nio-8080-exec-1] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/depends/api] threw exception [Request processing failed; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Access denied for user 'dependsapp'@'127.0.0.1' (using password: YES)] with root cause
java.sql.SQLException: Access denied for user 'dependsapp'@'127.0.0.1' (using password: YES)
/opt/mdhlabs #
Here is the part that has me stumped. This isn't a connectivity problem from within the container to the outer host that is running MariaDB and it isn't an authentication problem with the userid/password of dependsapp/badpassword. If I use the mysql client installed within the Docker container with the credential, the database can be accessed.
/opt/mdhlabs # mysql --user=dependsapp --password=badpassword --host=192.168.99.10 depends
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 45
Server version: 10.3.8-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [depends]> select project_id, projectname from projects where project_id=34;
+------------+------------------------+
| project_id | projectname |
+------------+------------------------+
| 34 | Ent Portal Unification |
+------------+------------------------+
1 row in set (0.000 sec)
MariaDB [depends]>
Here are the details of the exception logs generated within the Dockerized version of the SpringBoot app when it attempts to connect.
2019-08-24 18:56:14.605 INFO 1 --- [http-nio-8080-exec-1] c.c.depends.services.ProjectController : QUERY action=projectRe
2019-08-24 18:56:14.788 ERROR 1 --- [http-nio-8080-exec-1] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initi
java.sql.SQLException: Access denied for user 'dependsapp'@'127.0.0.1' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710) ~[mysql-connector-java-5.1.
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2205) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2236) ~[mysql-connector-java-5.1.41.jar!/:5.1
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2035) ~[mysql-connector-java-5.1.41.jar!/:5.1.41]
I am assuming that an internal SpringBoot JVM inside a Docker container and an internal mysql client binary inside a Docker container will both appear to be coming from the same source IP when a connection request hits the external MariaDB database. If it works from the mysql client inside the Docker container, that same userid/password should be allowed to connect when used inside a SpringBoot JVM in that same Docker container.
The same problem occurs with OpenJDK 12 and openjdk:12-alpine so I don't think the issue is version related in Java. Could the mysql-connector JAR (version 5.1.41) have an issue with the newer versions of Java or with MariaDB 10.3.x?
Any help would be appreciated.
