I have Apache 2.2.24 serving as a reverse proxy for 2 Tomcat 7.0.35 instances running on JDK 1.6.0_17. The Tomcats run a BlazeDS webapp that serves up content to an Adobe AIR App.
So the config looks like
Adobe AIR App ---HTTPS----> Apache ---HTTPS---> Tomcat
Due to infrastructure standards in my company, the Apache connects to Tomcat over SSL and uses HTTP instead of AJP. This can't be changed.
The Apache config looks like this:
<IfModule mod_ssl.c>
<VirtualHost xxx:461>
SSLEngine on
# Force to use explicit Ciphers first (see artf660612)
SSLHonorCipherOrder on
# Disable SSLv2
SSLProtocol -ALL -SSLv2 +SSLv3 +TLSv1
# Disable Weak Ciphers
SSLCipherSuite !aNULL:!eNULL:!EXPORT:!DSS:!DES:RC4-SHA:RC4-MD5:+HIGH:+MEDIUM:ALL
SSLCertificateKeyFile /vhosts/content/current/contentsys/ssl/ssl.key
SSLCertificateFile /vhosts/content/current/contentsys/ssl/ssl.cert
SSLCACertificateFile /apache22/conf/ssl.crt/ca-bundle.crt
<FilesMatch "\.(cgi|shtml|pl|phtml|php3?)$">
SSLOptions +StdEnvVars
</FilesMatch>
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
# Add per-virtual host throttling
<IfModule mod_qos.c>
# Defines the maximum number of concurrent TCP connections for this server (virtual host).
QS_SrvMaxConn 50
</IfModule>
<IfModule mod_cband.c>
CBandScoreboard /var/run/http/apache22/cbandscore/dw12869
# ridiculously high bandwidth and requests per second as we are only
# concerned with the last parameter - concurrent requests
CBandSpeed 100M 1000 50
</IfModule>
<LocationMatch "/(WEB|META)-INF/">
AllowOverride None
order allow,deny
deny from all
</LocationMatch>
# Import global URL Rewriting Engine properties
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteOptions inherit
</IfModule>
## Workaround for Apache "Range:" HTTP header bug (CVE-2011-3192)
# Drop the Range header when more than 5 ranges.
SetEnvIf Range (?:,.*?){5,5} bad-range=1
RequestHeader unset Range env=bad-range
# We always drop Request-Range; as this is a legacy
# dating back to MSIE3 and Netscape 2 and 3.
RequestHeader unset Request-Range
## END OF: Workaround for Apache "Range:" HTTP header bug (CVE-2011-3192)
## Workaround for PHP CGI remote code disclosure and execution bug (CVE-2012-1823)
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{QUERY_STRING} ^(%2d|-)[^=]+$ [NC]
RewriteRule ^(.*) $1? [L]
</IfModule>
## END OF: Workaround for PHP CGI remote code disclosure and execution bug (CVE-2012-1823)
# mod_jk configuration
<IfDefine JK>
<IfModule mod_jk.c>
</IfModule>
</IfDefine>
<IfModule ct_apache_mod.c>
RewriteEngine on
RewriteRule ^/(healthcheck|web-info) - [S=1]
# restriction: if the HOST header is not on the allowed list, serve a redirect to canonical URL
</IfModule>
#
# BalancerConfiguration 11
#
<Proxy balancer://balancer11>
BalancerMember https://myserver1.com:19802 route=0 timeout=300
BalancerMember https://myserver2.com:19802 route=1 timeout=300
ProxySet stickysession=JSESSIONID|jsessionid
ProxySet scolonpathdelim=On
ProxySet lbmethod=byrequests
</Proxy>
<Location /myapp>
ProxyPassReverse balancer://balancer11/myapp
ProxyPassReverseCookiePath /myapp /myapp
ProxyPass balancer://balancer11/myapp
</Location>
# BEGIN Extra configuration verbatim from config file
# Cache ADK CSS and Image Assets
<Location /adk>
ExpiresActive On
ExpiresDefault "access plus 1 year"
</Location>
LoadModule deflate_module /apache22/modules/mod_deflate.so
<IfModule mod_deflate.c>
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|bmp|icon|ico|tif|tiff)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
# END Extra configuration verbatim from config file
</VirtualHost>
</IfModule>
Tomcat Server.xml looks like
<Connector protocol="org.apache.coyote.http11.Http11Protocol" port="19802"
maxThreads="200" connectionTimeout="20000" scheme="https" secure="true" SSLEnabled="true"
keystoreFile="/path/to/keystore" keystorePass="changeIt" keyAlias="web_ssl"
clientAuth="false" sslProtocol="TLS"/>
The tomcat server restarts everyday at 12.45 am. We have about 20 users logging in starting 8 am. The flex app uses polling so makes a HTTP call every 1 second to fetch data. Everything works fine for the first few hours of the day. Then about 3-4 pm, we start getting 502 Bad Gateway errors sporadically on calls to 1 Tomcat Node.
[Fri Aug 16 20:01:19 2013] [info] [client xx.xx.xx.xxx] (70007)The timeout specified has expired: SSL input filter read failed.
[Fri Aug 16 20:01:19 2013] [error] [client aa.aa.aa.aaa] (70007)The timeout specified has expired: proxy: error reading status line from remote server myserver1.com:19802, referer: app:/Main.swf
[Fri Aug 16 20:01:19 2013] [error] [client aa.aa.aa.aaa] proxy: Error reading from remote server returned by /myapp/messagebroker/amfsecure, referer: app:/Main.swf
Sometimes it's the following
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (104)Connection reset by peer: SSL input filter read failed.
[Fri Aug 16 20:17:30 2013] [error] (104)Connection reset by peer: proxy: prefetch request body failed to yy.yy.yy.yy:19802 (myserver.com) from xx.xx.xx.xx ()
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (104)Connection reset by peer: SSL input filter read failed.
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Aug 16 20:17:30 2013] [info] [client xx.xx.xx.xxx] Connection closed to child 26 with standard shutdown (server mysite:443)
We have sticky sessions set so new users attempting to log in get repeatedly directed to 1 node which could be one throwing this error and so they keep getting 502 timeouts on restsarting as well. Restarting the Tomcat node fixes the problem and it starts working for a few hours. But then it re-appears again. The tomcat itself is directly accessible without issues. One other thing to note is that the Apache server also acts as a reverse proxy for other Tomcats that I don't manage/control.
I've been searching for other folks facing this problem and found the following:
- Chunking with GZIP on SSL reverse proxy is known to have issues - https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
- People have pointed out that setting force-proxy-request-1.0 & proxy-nokeepalive can help - https://serverfault.com/questions/206738/intermittent-error-when-using-mod-proxy-to-do-reverse-proxy-to-soap-service
We can't edit the Apache file directly as it's a controlled env and can only add config options at the bottom. So I added the following
SetEnvIf Request_URI "/myapp.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
I thought it would help as load testing with JMeter in UAT looked promising. The log below shows the output for 1 such request after the above directive was added. However, we're still seeing this issue in production
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(46): proxy: BALANCER: canonicalising URL //balancer11/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(1771): proxy: start body send
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(280): proxy: BALANCER: Found value B8620734D4F2D92E9B0C507C16A5EF8A.1 for stickysession JSESSIONID|jsessionid
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(290): proxy: BALANCER: Found route 1
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(594): proxy: BALANCER (balancer://balancer11) worker (https://myserver1.com:19802) rewritten to https://myserver1.com:19802/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy.c(1028): Running scheme balancer handler (attempt 0)
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(2001): proxy: HTTP: serving URL https://myserver1.com:19802/myapp/messagebroker/amfsecure
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2013): proxy: HTTPS: has acquired connection for (myserver1.com)
[Wed Aug 21 15:57:07 2013] [debug] ssl_engine_io.c(1918): OpenSSL: I/O error, 5 bytes expected to read on BIO#cfa3578 [mem: d1f4dd8]
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2069): proxy: connecting https://myserver1.com:19802/myapp/messagebroker/amfsecure to myserver1.com:19802
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2195): proxy: connected /myapp/messagebroker/amfsecure to myserver1.com:19802
[Wed Aug 21 15:57:07 2013] [debug] mod_deflate.c(615): [client xx.xx.xx.xxx] Zlib: Compressed 0 to 2 : URL /myapp/messagebroker/amfsecure, referer: app:/Main.swf
[Wed Aug 21 15:57:07 2013] [debug] mod_headers.c(756): headers: ap_headers_output_filter()
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_http.c(1875): proxy: end body send
[Wed Aug 21 15:57:07 2013] [debug] proxy_util.c(2031): proxy: HTTPS: has released connection for (myserver1.com)
[Wed Aug 21 15:57:07 2013] [debug] mod_proxy_balancer.c(637): proxy_balancer_post_request for (balancer://balancer11)
[Wed Aug 21 15:57:07 2013] [info] [client xx.xx.xx.xxx] Connection closed to child 15 with unclean shutdown (server myserver:443)
All similar issues reported by users seem to have occurred on earlier versions of Apache, Tomcat etc and the bug's long been fixed. So I really don't know what to try/do. Any pointers will be appreciated. Thanks
