Possible for HAProxy to perform a mix of SSL termination and passthrough?

Question

I'm working on analyzing an intermittent load balancing failure in our application.

Previously we had been using the AWS Elastic Load Balancer in the following configuration:

HTTP 80 -> HTTP 80
HTTPS 443 -> HTTPS 443, presenting the same certificate as the backend IIS servers are

Now we realize this is a bit of a hairbrained setup (both the LB & the IIS server doing the same encryption ... wasted work), and it has now started to cause some problems for us. Specifically, we will intermittently see a request to the ELB spike in latency, take a full 60s (the default timeout), and then report an error to the client. We spent a good deal of time confirming the latency spike is not related to a processing delay in the application.

As mentioned, we now realize this setup is weird. For example, a more natural config of ELB works just fine:

HTTP 80 -> HTTP 80
TCP 443 -> TCP 443, straight passthrough, all encryption happening on the IIS backend

Zooming out for a moment, we became curious if we could reproduce the intermittent failure in the bad configuration on HAProxy. That is have HAProxy do SSL termination, and then initiate another full SSL connection to the backend server. Again, we realize this is silly, but are investigating for purposes of comparing the black-ish box that is ELB and HAProxy.

Here is the simple config I tried:

frontend https_frontend
  bind *:443 ssl crt /etc/ssl/certs/ourpublicandprivatecert.pem
  mode http
  default_backend web_server
backend web_server
  mode http
  server s1 10.0.1.4:443 check

Then when browsing to the HAProxy host, we get:

504 Gateway Time-out

The server didn't respond in time.

I'm guessing HAProxy is freaking out about the mismatch on the cert, but I can't get logs to confirm that. The other possibility is that because that is such a weird setup (mixing SSL termination and passthrough) HAProxy just doesn't support it, instead forcing you down the much saner paths of passthrough OR termination.

Anyone have any insight?

score 1 · Answer 1 · edited Mar 31 '15 at 06:50

Naturally you have a recent version of HAProxy with OpenSSL support built in.

It seems you need some extra parameters to use HTTPS backend. HAProxy tries normal HTTP connection by default, regardless of the port number.

The ssl parameter ensures SSL connection:

server s1 10.0.1.4:443 ssl check check-ssl

The server certificate is not verified by default. If you wish to verify it, you need to add verify required and ca-file <cafile> to the server line.

See more about the options here: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.2

hookenz · Answer 2 · 2015-03-31T07:13:09.323

I think there is a mistake in your backend config. Try instead:

frontend https_frontend
  bind *:443 ssl crt /etc/ssl/certs/ourpublicandprivatecert.pem
  mode http
  default_backend web_server
backend web_server
  mode http
  server s1 10.0.1.4:80 check

The changing being the port! And also make sure that your webserver is not redirecting from port 80 to 443 or that will confuse things.

One advantage you get when using haproxy or ELB to do the the SSL termination is that it can insert headers such as X-Forwarded-For.

If you're simply passing through SSL using the tcp option and not doing SSL on the ELB or with HAProxy there is no way to insert the headers because the stream is encrypted. Therefore there is no way for the webserver to do something intelligent with the request and all client connections in the webserver logs will look like they came from the elb or haproxy server IP!

So don't delegate encryption to the backend, instead do it all on the ELB or an HAProxy instance and make use of X-Forwarded-For.

Possible for HAProxy to perform a mix of SSL termination and passthrough?

2 Answers2