Java container that works as a Pod not working in a Job

Question

I'm trying to use a container that contains a Java tool to do some DB migrations on a MySQL database in a Kubernetes Job.

When I run the container locally in Docker (using a MySQL container in the same network), the tool runs as expected. And if I create a Pod using the container and set the command arguments to point to the mysql service running in the same namespace, it does as well.

But if I convert that Pod spec into a Job, the created container can not connect to the MySQL service anymore for some reason.

The container is based on amazoncorretto:8-al2-jdk and just copies the JAR to /opt/.

The MySQL DB is available through the mysql service in the cluster:

$ kubectl describe service mysql -n <namespace>
Name:              mysql
Namespace:         <namespace>
Labels:            app=mysql
Annotations:       <none>
Selector:          app=mysql
Type:              ClusterIP
IP Families:       <none>
IP:                <ip>
IPs:               <ip>
Port:              mysql  3306/TCP
TargetPort:        3306/TCP
Endpoints:         <ip>:3306
Session Affinity:  None
Events:            <none>

These are the specifications for the Pod:

apiVersion: v1
kind: Pod
metadata:
  name: java-tool-pod
spec:
  containers:
  - name: javatool
    image: <registry>/<image-name>:<version>
    command: [ "/bin/sh" ]
    args: [ "-x", "-c", "/usr/bin/java -jar /opt/<tool>.jar \"jdbc:mysql://mysql:3306/<db>\" -u <user> -p<password>" ]
  imagePullSecrets:
  - name: <secret>

Running the Container as a Pod:

$ kubectl apply -f /tmp/as-pod.yaml -n <namespace>
pod/java-tool-pod created

$ kubectl logs pod/java-tool-pod -n <namespace>
+ /usr/bin/java -jar /opt/<tool>.jar jdbc:mysql://mysql:3306/<db> -u <user> -p<password>
DB Migration Tool
Database Schema, 3.30.0.3300024390, built Wed Jul 14 12:13:52 UTC 2021
Driver class: com.mysql.jdbc.Driver
INFO Flyway 3.2.1 by Boxfuse
INFO Database: jdbc:mysql://mysql:3306/<db> (MySQL 5.7)
INFO Validated 721 migrations (execution time 00:00.253s)
INFO Current version of schema `<db>`: 3.29.0.10859.10
WARN outOfOrder mode is active. Migration of schema `<db>` may not be reproducible.
INFO Schema `<db>` is up to date. No migration necessary.

These are the specifications for the Job:

$ cat /tmp/as-job.yaml 
apiVersion: batch/v1
kind: Job
metadata:
  name: javatool-job
spec:
  template:
    spec:
      containers:
      - name: javatool
        image: <registry>/<image-name>:<version>
        command: [ "/bin/sh" ]
        args: [ "-x", "-c", "/usr/bin/java -jar /opt/<tool>.jar \"jdbc:mysql://mysql:3306/<db>\" -u <user -p<password>" ]
      imagePullSecrets:
      - name: <secret>
      restartPolicy: Never

Running the container as a Job:

$ kubectl apply -f /tmp/as-job.yaml -n <namespace>
job.batch/javatool-job created

$ kubectl logs job.batch/javatool-job -n <namespace>
+ /usr/bin/java -jar /opt/<tool>.jar jdbc:mysql://mysql:3306/<db> -u <user> -p<password>
DB Migration Tool
Database Schema, 3.30.0.3300024390, built Wed Jul 14 12:13:52 UTC 2021
Driver class: com.mysql.jdbc.Driver
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:983)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:339)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2252)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2285)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2084)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:208)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
    at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:153)
    at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
    at com.nordija.itv.db.FlywayMigrationSchemaData.isNotFlywaySchemaVersion(FlywayMigrationSchemaData.java:58)
[...]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:214)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:298)
    ... 22 more
INFO Flyway 3.2.1 by Boxfuse
Unable to obtain Jdbc connection from DataSource
[...]

I haven't seen any significant differences in the containers being created. The only thing I can think of is some kind of character encoding issue, but I don't see why that should only occur in a Pod that was created for a Job and not in one that was created directly.

Thanks in advance for any help with this issue!

Edit: I forgot to mention that Istio is active on the Namespace, which turned out to be causing the issues.

Yes, the credentials were correct (they were copy & pasted). It turned out to be a problem related to Istio (which I forgot to mention is active in the namespace). — wtfc63, Aug 24 '21 at 11:53

score 1 · Accepted Answer · answered Aug 24 '21 at 11:50

The problem was that Istio doesn't play nice with Kubernetes Jobs (I forgot to mention that Istio is active on the Namespace, sorry).

Once I added a short delay (sleep 5 before starting the Java tool), the connection could be established.

But then I had another issue: After the container terminated successfully, the Job would not be marked as completed.

And the reason was again Istio. Jobs are considered complete once all Pods are terminated and the Istio Sidecar is a service Pod that doesn't terminate. After finding this article, I ended up integrating their scuttle tool into the container and now the Job can be completed successfully.

score 0 · Answer 2 · answered Aug 21 '21 at 07:20

0

The error message is "Communications link failure". So I think you should have a look at your Networkpolicies.

answered Aug 21 '21 at 07:20

weranders

176
2

Thanks for the tip, it didn't end up being the Network Policies, but Istio delaying the availability of the correct network routing. – wtfc63 Aug 24 '21 at 11:51

Java container that works as a Pod not working in a Job

2 Answers2