3

We have a SaaS application that allows customers to use their own domain – pretty typical. At its core, the architecture has a simple catch-all virtual host (apache) and the software does the work of resolving which customer it is based on the hostname.

However, many of our customers require that custom domain to support SSL. Currently we do this by adding a (higher precedence) virtual host with an SSL certificate and restarting apache.

While this works, it has a number of issues:

  1. You either have to restart apache which means momentary downtown, or switch the dns to an entirely new server stack.

  2. On the off chance there is an issue we didn't catch with the SSL certificate, Apache throws a fatal error and will not be able to restart.

  3. It would be impossible or at least impractical to roll this out to thousands of users.

In short – it doesn't scale. We've explored SSL termination at the load balancer...it's promising, but we can't find any load balancers that support multiple SSL certificates (via SNI).

Moving forward this is increasingly a bottle neck of the product – to the point we would be willing to alter architecture or move away from apache if needed.

jpschroeder
  • 143
  • 5
  • 1
    More than one server behind a load balancer, with rolling restarts. – ceejayoz Jun 14 '16 at 14:34
  • Sure, that falls into the drawback #1 category though – something we're trying not to do. The spirit of the question is how can we install these things without a major update (like a restart). Restarting servers for 1000s of users and having them flow in/out of load balancers seems like a serious issue. – jpschroeder Jun 14 '16 at 15:08
  • 1
    You said "we would be willing to alter architecture". Restarting servers used by thousands of concurrent users is a solved problem. – ceejayoz Jun 14 '16 at 15:10
  • Interesting. I suppose I could see that working (+1). Similarly we could spin up new servers with the SSL installed and dropping old servers out...could work. Our plan is to have SSL installation be a feature the customers perform themselves, which gives me some pause to think their actions directly affect server count - maybe an unnecessary fear. So let me change the question slightly then to keep digging – are there any technologies that allow "hot" installation of ssl certificates without the need for *anything* to restart? – jpschroeder Jun 14 '16 at 15:46

2 Answers2

2

We've explored SSL termination at the load balancer...it's promising, but we can't find any load balancers that support multiple SSL certificates (via SNI)

Really? I had to double take on the post time of this question to make sure it wasn't a years old necro-thread because I can't think of any load balancers that don't support SNI in 2016.

Though the one I'm most familiar with is HAProxy which has supported SNI since 1.5 which apparently came out in June 2014.

While 1.5 doesn't technically support true zero downtime reloads for config changes, it has a "fast reload" option that is generally fast enough to not impact a typical site. But if you really want to get there, there's a Yelp Engineering blog post describing a method to do "true zero downtime" reloads that involves some fancy work with Linux tc and iptables to delay SYN packets during a reload. From the article:

Restarting HAProxy has basically no effect on our traffic, causing only minor delays as can be seen in Figure 3. Note that this method is heavily dependent on how long HAProxy takes to load its configuration, and because we are running such a reduced configuration, these results are deceivingly fast. In our production environment we do observe about a 20ms penalty during HAProxy restarts.

Ryan Bolger
  • 16,755
  • 4
  • 42
  • 64
  • This is great to know, I'm familiar with cloud balancers such as Rackspace and Amazon's ELB – neither of which currently support SNI. You're making a good case for having home-rolled one though. – jpschroeder Jun 15 '16 at 19:09
  • Hah, I didn't even consider the case that you guys might be hosting your SaaS app in a public cloud environment. I'm definitely not as well versed in the LBaaS options. – Ryan Bolger Jun 15 '16 at 19:16
1

A load balancer means restarting a drained server is fine and not something to be avoided.

The customer could request a new domain or certificate. On the next batch of updates, this change is in a new config is deployed to some web servers. Drain the old web servers and stop them at zero connections. New connections are going to the new working config. If there is a problem, revert to the last config.

Add web servers to the load balancer for performance and availability reasons. Add load balancers when the complexity of the config is too much.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34