1

We are moving a production environment to a customer's data centre. At the moment, for the purpose of explaining the scenario, we have an Apache Load Balancer that is load-balancing requests to 2 Apache Tomcat application servers that are hosting 2 of our grails applications. I'd appreciate any help!

To summarize, our setup is: - An Apache HTTP Server acting as a Load Balancer

  • 2 Tomcat Servers (Node A and Node B)

  • We have two Java/Grails applications being hosted on each of the Tomcat servers. On our current environment we send hessian requests from one of our grails applications to another grails application, via our load balancer. This works perfectly. On the environment we're migrating to, this is failing.

    We've been able to verify all firewalls and security packages are turned off. We've also been able to verify there is an error in the load balancer access log everytime this hessian call fails (/var/log/httpd/access_log) :

10.30.95.51 - - [19/Jun/2014:11:55:13 -0400] "POST /Application/hessian/RemoteTicketService HTTP/1.1" 500 8067 "-" "Java/1.6.0_30"

I went ahead and performed a tcpdump and I was able to determine that there is a Grails Application error being returned for Hessian - does anyone have any ideas?:

L6DàL6<83>¢HTTP/1.1 500 Internal Server Error^M Date: Fri, 20 Jun 2014 14:40:08 GMT^M Server: Apache/2.2.15 (Red Hat)^M Connection: close^M Transfer-Encoding: chunked^M Content-Type: text/html;charset=UTF-8^M ^M 1f83^M Grails Runtime Exception .message { border: 1px solid black; padding: 10px; background-color:#E9E9E9; margin: 30px; } .header{ margin: 30px; } .stack { border: 1px solid black; padding: 5px; overflow:auto; height: 300px; } .snippet { padding: 5px; background-color:white; border:1px solid black; margin:3px; font-family:courier; }

Looks like you've encountered an error, sorry about that Message: Expected 'H'/'C' (Hessian 2.0) or 'c' (Hessian 1.0) in hessian input at -1
Caused by: Hessian skeleton invocation failed; nested exception is java.io.IOException: Expected 'H'/'C' (Hessian 2.0) or 'c' (Hessian 1.0) in hessian input at -1 Class: Unknown
At Line: [-1]
Code Snippet:

dmcgee
  • 41
  • 1
  • 5

1 Answers1

1

This turned out to be an issue with the mod_jk module 1.2.39 on Apache/2.2.15. We swapped the mod_jk module out to version 1.2.37 and everything worked fine. Using tcpdump and a basic hessian java project, we were able to confirm the issue was resolved by setting "chunked post" to false on our hessian factory. Unfortunately, we couldn't perform this code change in our project libraries/dependencies, obviously, but we worked out that the forwarding of chunked HTTP requests was broken in mod_jk 1.2.39. mod_jk 1.2.40 fixed this bug and it doesn't seen to impact our environment in 1.2.37 either.

http://tomcat.apache.org/connectors-doc/miscellaneous/changelog.html

dmcgee
  • 41
  • 1
  • 5