I have a fairly simple Docker implementation that consists of two containers running on an overlay network: one running Apache HTTP 2.4.16 and another running a Spring Boot app (using Tomcat 8). The Apache container handles all external https requests and proxies traffic (via mod_proxy_http) to the Spring Boot app running in the second container. My proxy configs look like this:
ProxyPass /web-portal http://web-portal:8080/web-portal
retry=0
ProxyPassReverse /web-portal http://web-portal:8080/web-
portal
ProxyTimeout 300
I'm setting the Spring Boot/Tomcat server.connection-timeout to the same value as Apache's ProxyTimeout
This all works just fine when running in Docker; however, I'm wanting to move to Docker Swarm and that's when I run into issues. I've set up a very simple Docker swarm environment with a single node that also serves as the manager (by running docker swarm init). I am running the same exact containers that I describe above. Initially, behavior is as expected: Apache proxies https traffic destined to /web-portal to the Spring Boot app. After a period of time (that is seemingly unpredictable) the proxy stops working and I see entries like this in the Apache ssl_error.log file:
[proxy_http:error] [pid 222] (70007)The timeout specified has expired:
[client 10.255.0.2:64842] AH01102: error reading status line from
remote server web-portal:8080, referer:
https://ojbc.org/web-portal/
[proxy:error] [pid 222] [client
10.255.0.2:64842] AH00898: Error reading from remote server returned by
/web-portal/activity, referer: https://ojbc.org/web-
portal/
When this happens, the only solutions I've found are to docker rm -f the Apache HTTP container and create a new container (docker run) or just wait for some period of time and the proxy starts working again.
To eliminate variables, I removed Spring Boot from the equation and just ran plain ol' Tomcat8 in the second container, but the issues were still present. I've also tried proxying via mod_proxy_ajp rather than mod_proxy_http -- but I get similar errors.
I have never seen these problems when running these containers in normal (non-Swarm) Docker mode. I've scoured this site, and others, trying to better understand why introducing Docker Swarm would cause these issues and what the solution might be, but I'm at a loss. I'm guessing it might have something to do with load balancing in Docker Swarm, but I don't have much to back up that theory. I appreciate any thoughts you may have. Please let me know if I can provide any further detail about my environment.
