7

I am trying to create docker containers for ZooKeeper and configure them in cluster mode (full code is here and here).

Containers are based on Alpine Linux (alpine:3.2 on Docker Hub), but the problem that I'm going to describe happens also with the official Java container (java:7).

I use the following commands to start the cluster: 

docker run -d -h zk1 --name zk1 dockmob/zookeeper -s zk1,zk2,zk3
# wait some time ...
docker run -d -h zk2 --name zk2 dockmob/zookeeper -s zk1,zk2,zk3
docker run -d -h zk3 --name zk3 dockmob/zookeeper -s zk1,zk2,zk3

(They are available on docker hub, you can try them).

If I wait some time before starting the second and third containers, then the host names zk2 and zk3 are put in /etc/hosts too late (by docker), and Java is unable to find them: I get java.net.UnknownHostException in the logs of zk1 for both zk2 and zk3.

I found on the web that I need to disable JVM DNS cache in order to refresh the host names, so I introduced the following command in the Dockerfile in order to update the java.security settings:

RUN grep '^networkaddress.cache.ttl=' /usr/lib/jvm/java-1.7-openjdk/jre/lib/security/java.security || echo 'networkaddress.cache.ttl=10' >> /usr/lib/jvm/java-1.7-openjdk/jre/lib/security/java.security

It sets the DNS TTL property (networkaddress.cache.ttl) to 10 seconds.

The variable networkaddress.cache.negative.ttl is already set to its default value (10).

The behavior does not change. I get lots of java.net.UnknownHostException repeatedly.

What can be the cause of the problem?

Nicola Ferraro
  • 4,051
  • 5
  • 28
  • 60
  • If I get this situation right: your first command introduces a new docker instance with hostname zk1, and instructs it to form a cluster with zk2 and zk3 ... THEN ... if you are not fast enough creating zk2 and zk3 by executing line #2 and #3, then line #1 will fail due to not being able to find the hosts for the next two cluster-members. Is this what you are experiencing? – Gergely Bacso Sep 12 '15 at 11:34
  • Almost. First node tries continuously to contact zk2 and zk3, but the java network stack always uses the old information about them (i.e. they don't exist). The new containers are put in /etc/hosts once they are created, but java seems to ignore the file. – Nicola Ferraro Sep 12 '15 at 11:39
  • The final aim is to allow attaching containers to the ZooKeeper cluster at runtime. I introduced the delay to start testing this situation. – Nicola Ferraro Sep 12 '15 at 11:45
  • I see. I think your idea is ok, but you are playing with the wrong variable. Since initially your DNS resolution fails, you should be disabling the negative cache. Check the current value for this variable (same java.security file): networkaddress.cache.negative.ttl . It might be on 10 by default, but if it is not, it is worth setting it to some low figure. – Gergely Bacso Sep 12 '15 at 11:50
  • You're right, but I checked the `networkaddress.cache.negative.ttl` and it is explicitly set to the default (10). – Nicola Ferraro Sep 12 '15 at 12:02
  • What is your exact OS and JDK version? – Gergely Bacso Sep 12 '15 at 12:10
  • It happens with docker java:7 container: `java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb8u1) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)`, but also Alpine Linux container (alpine:3.2) `java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.6) (Alpine 7.79.2.5.6-r0) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)` – Nicola Ferraro Sep 12 '15 at 12:42
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/89461/discussion-between-gergely-bacso-and-nicola-ferraro). – Gergely Bacso Sep 12 '15 at 13:33

2 Answers2

11

In my case the java application was failing with java.net.UnknownHostException when running in docker. The reason was that I used --network=none docker flag (getting ip/hostname via dhcp and pipework). In this case, docker does not add automatically to /etc/hosts entry like

127.0.0.1 15e326aecf84

And getCanonicalHostName() Java function threw this exception.

Possible solutions:

  • add hostname entry to /etc/hosts file via docker run parameter --hostname=your-hostname.com
  • switch to docker-managed network configuration
Matthias Hamann
  • 719
  • 8
  • 27
Dmitriusan
  • 11,525
  • 3
  • 38
  • 38
1

I managed to get rid of the DNS issues by switching to Oracle JRE 8 and using the following hack in the Dockerfile:

RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf

I created a working Java 8 docker container container on Docker Hub (the code is on github).

Nicola Ferraro
  • 4,051
  • 5
  • 28
  • 60