3

I have a Haskell application using Happstack that will run fine for anywhere from 1-5 days and then suddenly become unresponsive, with the log looking something like this:

ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:140A1175:SSL routines:ssl_bytes_to_cipher_list:inappropriate fallback"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"
ProtocolError "error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac"

Terminating and restarting the application gets it working then for another few days.

What are some possible causes of this situation, and/or what can I do to trace/debug this issue?

ryachza
  • 4,460
  • 18
  • 28
  • Interesting find. A friend of mine ran into this with a snap based server but didn't get to an answer and instead ran the TLS on another server and locally proxied the connection. Your exact situation is a little different - his problem was that every once in a while a connection would hang - leaving the client in limbo - and he would see a few of the same messages you are posting (in general the server still worked). – Thomas M. DuBuisson Nov 30 '15 at 19:54
  • My working theory is that the non-thread-safe (as in, not green thread safe due to thread local storage) values inside of OpenSSL and the manner in which the TLS support depends on OpenSSL is resulting in race conditions. If you go in and put a mutex around every openssl (foreign) call does the situation improve? (This was not practical last time I heard about the bug, but if it is practical for you then it would provide a small amount of additional data with which to continue debugging). – Thomas M. DuBuisson Nov 30 '15 at 19:55
  • @ThomasM.DuBuisson Thanks for your response. That's a shame that you've heard of this with another server since I was leaning towards just trying something other than Happstack. When you say "ran the TLS on another server..." do you mean like throwing something like nginx in front of it? Because that was another option I was considering. Currently I'm using `happstack-server-tls` so I'm not sure your idea about wrapping OpenSSL calls is practical but I will keep the thought in mind as I move forward. – ryachza Nov 30 '15 at 20:30
  • Yes, throwing nginx in front is exactly what I was talking about and should be a functional, though admittedly hackish, solution. – Thomas M. DuBuisson Nov 30 '15 at 20:47

0 Answers0