0

We've been using NGINX as cache layer and recently when load testing it we've stumbled upon an unusual problem: the load test would only get cache miss on all the content.

The traffic flows like this: Client > NLB > NGINX > ELB (stickiness) > Application Layer

After a bit of analysis and troubleshooting, I've come to the conclusion that the reason for this behaviour is the lack of the "AWSELB" stickiness cookie in the requests our load injector is sending. When the cookie isn't set, the AWS ELB will send "set-cookie" with a value for AWSELB and will also include a cache-control: no-cache="set-cookie".

Since it has a "no-cache" directive, NGINX ends up not creating the expected cache entry unless that cookie is sent (later requests after this point will have cache HIT behaviour). Been going through a lot of links/questions and the documentation but I've been unable to find a solution so far. Has anyone gone through this and how have you handled it?

Sample request:

GET /asset/style.1234567.css HTTP/2
Host: mytestserver
User-Agent: curl/7.54.0
Accept: */*

Sample response snippet:

< HTTP/2 200 
< server: nginx
< date: Tue, 04 Dec 2018 16:35:50 GMT
< content-type: text/css; charset=UTF-8
< content-length: 572353
< accept-ranges: bytes
< cache-control: public, max-age=3600
< cache-control: no-cache="set-cookie"
< set-cookie: AWSELB=VERYBIGCOOKIEWITHAWSELBDETAILS;PATH=/;MAX-AGE=60
< x-proxy-cache: MISS

Closest post/article I've seen to my problem signature but applies to CloudFront and hasn't got any details on how it was handled: https://forums.aws.amazon.com/thread.jspa?messageID=362272

bayetovsky
  • 115
  • 10
  • *"Since it has a "no-cache" directive, NGINX ends up not creating the expected cache entry unless that cookie is sent"* ...wait. That is a *response* header. Nginx behind the ELB would not see it. You are talking about Nginx behind the ELB, right? Or is it in front? – Michael - sqlbot Dec 05 '18 at 16:31
  • @Michael-sqlbot the ELB is in front of the application (and after the nginx). I'm including the traffic flow in the original post as this made me notice it was unclear. The traffic flow is Client > NLB > NGINX > ELB (stickiness) > Application Layer – bayetovsky Dec 05 '18 at 17:33
  • Stickiness and cacheability seem like opposing objectives (to me). Speaking conceptually, stripping those headers for cacheable resources seems like the objective. Dirty solution: two ELBs (one sticky, one not) and use Nginx config to route requests requiring stickiness to one ELB and others to the other (path based?). Another solution, caching with CloudFront instead of Nginx, using Lambda@Edge Origin Response triggers to strip the headers in appropriate cases. Or maybe something in Nginx+Lua to hook and scrub responses on the back side of the cache. Or Nginx > HAProxy (scrubbing) > ELB. – Michael - sqlbot Dec 05 '18 at 18:56
  • There are a lot of possible permutations. If (by chance) you are using the NLB to get static external addresses, you can now get those with an ELB using [Global Accelerator](https://aws.amazon.com/blogs/aws/new-aws-global-accelerator-for-availability-and-performance/) and you could use GA > ELB > Nginx > App. Or avoid stickiness, of course, which I do with DynamoDB. – Michael - sqlbot Dec 05 '18 at 19:01
  • the upstream server is actually the entry point to several applications and one of them requires the stickiness. Having the static assets (images, js, css, etc) served from a cache server doesn't mean that there shouldn't be other requests that go through that require stickiness (if this makes any sense the way I explained it just now). Regardless of that, the "No-Cache=Set-Cookie" should apply only to not caching the mentioned header and not the whole response – bayetovsky Dec 05 '18 at 21:24

0 Answers0