2

I have an app running in a custom Java container on App Engine flexible. I've been trying to track down why I've been getting stale responses, and I narrowed the issue down to responses being cached.

I know our app isn't returning the stale content, because we have no shared cache, and stale responses persist after we redeploy. We also aren't using Cloud CDN.

When I hit my app's API at https://[app-id].appspot.com/api/123, my app returns a Cache-Control: public, s-maxage=31536000, max-age=3600 response header. s-maxage was intended to tell Cloudflare (our CDN) to cache the response, and we manually purge the Cloudflare's cache when we update content. However, it appears that something in front of our app is caching responses. In the cached responses, the date response header stays the same and the age increases, showing that the content is indeed stale.

I've seen documentation regarding caching in the App Engine standard docs for app.yaml, but I couldn't find any mention of caching infrastructure in the App Engine flexible docs. My mental model was that only HTTPS load balancing was happening on behalf of the app, but that doesn't appear to be the case.

If I try to reproduce this issue from within GCP by SSHing into a Compute Engine instance or App Engine Flexible instance, I don't observe any caching. However, I've tried this from other clouds and laptop and the caching is consistent. Cloudflare support was who originally alerted us to this behavior, and they're seeing the caching from their production data centers, which are peered directly with Google via their CDN Interconnect program.

This must be coming from Google's infrastructure itself, any maybe the hidden Google Cloud Load Balancer used for App Engine, since the connection is encrypted.

What's doing the caching, how do I disable it, and how do I purge it?

jon_wu
  • 1,113
  • 11
  • 26
  • Suggestion: Stop changing content. Use version numbers or hash values for new versions of static files in the filename. There are so many variables (and devices, caches, proxies) you do not control in caching. – John Hanley Dec 03 '19 at 02:36
  • Generally that's our goal, but for some things it's not an option or is too difficult to sync up the version numbers everywhere. Instead we can rely on our CDN's cache to keep content until we tell it to purge, but have all clients refresh from the CDN regularly. Agreed that we can't make many guarantees about caches in general, but from the origin up unto the CDN, we _should_ be able to know exactly what's going on. – jon_wu Dec 03 '19 at 04:37
  • Agreed. Interesting question. – John Hanley Dec 03 '19 at 05:11
  • After thinking a little more, I don't think there should be anything (unexpected) we can't control, beyond this hidden behavior on Google's side - at least not if using `curl`. I agree that browsers can be a little unpredictable, but intermediate caches and proxies shouldn't be a factor here since we're using HTTPS and nothing in the middle can read the HTTP headers or content. – jon_wu Dec 04 '19 at 22:46
  • Very good comment "nothing in the middle". That leaves your code and the browser. Which one do you think is breaking the cache chain? We are back to my first comment. – John Hanley Dec 05 '19 at 00:08

1 Answers1

0

The serving infrastructure in front of your App Engine instances can potentially cache requests though this is not an officially documented feature. You're more likely to see this if you have a lot of clients hitting App Engine from the same network, resulting in all of them hitting the same group of caches and keeping those caches warm. There is no API to flush the cache so the suggestion provided in the comments above by John Hanley is the best mitigation: use version-specific URLs for your static files that you only want to be served by a specific version of your app.

John L
  • 26
  • 1
  • 1
    I think it's officially documented now (or maybe this isn't as detailed as you'd want): https://cloud.google.com/appengine/docs/standard/python3/how-requests-are-handled#response_caching – aldel Apr 18 '22 at 18:28