7

I'm building an application with some rather arcane TLS and caching requirements, and I'm using a stack that looks like:

     HTTPS             HTTP
  Nginx (443)       Varnish (80)
       |                |
  Varnish (80)      Nginx (8080)
       \               /
          AWS ALB (80)
               |
  ECS Pool (Docker) - Apache/80

(I have Nginx listening on 8080 in between Varnish and the ALB because Varnish doesn't handle DNS lookups with multiple/changing IP addresses as easily as Nginx).

The problem is the HTTPS request path: basically, Nginx sets X-Forwarded-Proto: https, and then I have my Varnish VCL configured to securely pass this through to the ALB. But the ALB seems to strip the header and replace it with its own (which becomes X-Forwarded-Proto: http), and then the backend application on the ECS servers sees http and writes all it's links/resource paths as http, causing an insecure mixed content warning in Safari, Chrome, etc.

So the only way I've been able to get this setup working is to bypass Varnish, install a backend-only TLS cert on the ALB, and set up request routing like:

     HTTPS             HTTP
  Nginx (443)       Varnish (80)
       |                |
 AWS ALB (443)      AWS ALB (80)
       \               /
        \             /
  ECS Pool (Docker) - Apache/80

I'm mostly okay with this... but it's a shame I'm not able to use Varnish to cache static resources for HTTPS requests. The only way around that, I imagine, would be to replace the ALB entirely with Varnish and a custom vmod or some other tooling (Lambda, etc.) to account for the ever-changing ECS Docker container instance pool...

Is there any way to command AWS ALBs/ELBs to pass through a header, or any kind of control over the headers? I couldn't find anything in the docs or Console, and most other people seem to be terminating HTTPS on the ALB. If I had just one cert, or a few, or could use SNI, I would definitely do that... but I have hundreds of certs for hundreds of domains, and can't :(

geerlingguy
  • 4,682
  • 8
  • 56
  • 92
  • An ELB (not ALB) in TCP (not HTTP) mode would pass this through unmodified. Since you have your own front-end (the balancer isn't the front-end) that seems like the simplest solution. Have you tried that? – Michael - sqlbot Jul 27 '17 at 22:34
  • I have not... the docs on ELB / ALB are quite a mess though—can I use an ELB in front of an ECS deployment (e.g. multiple tasks running in EC2 Container Services) and have the ELB work correctly with tasks/services changing instances after deployments/upsizes/downsizes? – geerlingguy Aug 01 '17 at 03:34
  • Sorry, I was focused on the proxy layers and overlooked the fact that you are working with ECS. In that case, stunnel4 or HAProxy on the Varnish nodes (after Varnish) to speak HTTPS to the inner ALB would do the trick, or using Varnish to add your own (made up name) `X-No-Seriously-The-Forwarded-Proto-Is: https` header, which ALB would pass through and Apache would be configured use this value to overwrite the incoming `X-Forwarded-Proto` if the alternate is present (before the application sees it). – Michael - sqlbot Aug 01 '17 at 09:28
  • Wait, there may be another reason why you want something between Varnish and ALB. I'm a bit out of the loop with current Varnish capabilities, but I can tell you that where I am using it, the source and destination from Varnish's perspective are *both* HAProxy -- HAProxy before and after. If your targets are ELB, ALB, S3, or CloudFront, the target is always specified by hostname, because the IP addresses can change. At the point where I deployed it, the capability of Varnish to update DNS lookups wasn't there, or wasn't clean, I don't recall the specifics. – Michael - sqlbot Aug 01 '17 at 09:39
  • @Michael-sqlbot - Yeah, I actually have Varnish pointing to Nginx on 8080, and Nginx routes the requests to the ALB. Sorry, will fix the diagram to reflect that. – geerlingguy Aug 01 '17 at 15:08
  • @Michael-sqlbot you are right, but things have changed recently, there is this vmod now: https://github.com/nigoroll/libvmod-dynamic – Razvan Grigore Aug 02 '17 at 22:35
  • If you have Nginx on 8080 talking to the ALB then you can just configure TLS on that segment. The cert doesn't matter -- you can sign your own for the ALB, and then give that exact same cert to Nginx to use as the ca-file. A certificate can authenticate itself on a private connection. – Michael - sqlbot Aug 02 '17 at 23:41
  • @Michael-sqlbot - Interesting, that may be an option here. Though it seems like it still may be a bit of extra work trying to make it so that port 80 requests from external traffic is still not routed through https. – geerlingguy Aug 03 '17 at 16:04
  • 2
    If it were me (well... if it were me, there'd be some HAProxy in here, but I digress) I'd add a special header with Nginx or Varnish and react to it with Apache, overriding X-Forwarded-Proro – Michael - sqlbot Aug 04 '17 at 03:34

0 Answers0