1

I have been experimenting with blue green deployment in kubernetes using nginx-ingress. I created few concurrent http request to hit v1 version of my application. In the meanwhile I switched the router to point to v2 version of my application. As expected v2 version was serving the requests after the swap ,but what made me curious is that all the request were success. It is highly probable that there were some in-progress request in v1 while I made the swap. Why didn't such request fail?

I also tried the same by introducing some delay in my service so that the request take longer time to process.Still none of the request failed.

bhavanak
  • 255
  • 1
  • 12
  • well, depends on your test. If you did couple of non-concurrent requests per seconds, that's candy for any server, probably it takes like 10 times more to switch from one version to another. Try sending concurrent requests; couple of hundred per second, and some of them will fail. – suren Jun 03 '20 at 19:05
  • I tried with 50 concurrent request 1000 times.Still all request are success. I doubt it nginx-ingress-controller does any graceful draining of request,but cant find any supporting document – bhavanak Jun 03 '20 at 19:24
  • 50 concurrent requests have not failed? that could have failed even with the server up. How are you sending these requests? Can you paste the command? – suren Jun 03 '20 at 19:26

1 Answers1

1

Usually in-flight requests are allowed to complete, just no new requests will be sent by the proxy.

coderanger
  • 52,400
  • 4
  • 52
  • 75
  • Is it some configuration in nginx-ingress controller that ensures in-flight requests are completed ?Because I noticed few request fail when i tried rolling upgrade. – bhavanak Jun 03 '20 at 18:09
  • by the way, there is no such thing as "allowing to complete" requests. A server goes down, that's it! Even if the request was right in front of the door, it is going to be dropped. – suren Jun 03 '20 at 19:07
  • Blue/green and rolling upgrade are not the same thing. In a blue/green, the old service stays up entirely, you just stop sending it traffic. In a rolling update things are being shut down as you go. K8s does remove the Endpoint before termination but it can take a bit for that to be reflected in the Ingress so you usually want to wait a few seconds in a pre-stop hook and then make sure to do a graceful shutdown of the web server (usually SIGTERM will do that by default but check). – coderanger Jun 03 '20 at 22:25
  • @suren There is a grade period between service deregistration and forced pod kill specifically to allow for things like in-flight requests completing. You are incorrect :) That grade period is even configurable and there is a hook for further customization. – coderanger Jun 03 '20 at 22:29