Google cloud storage returning older data

Question

How is this possible? That for the same file, the Date/Expires/Last-Modified headers are so dramatically different. Even though this filed was modified today (Apr 8), it randomly returns Apr 7 and after that Apr 3rd or 4th!

curl -I http://storage.googleapis.com/myappname.appspot.com/config/config.json
HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2UoPjuz3v6i-RioExWbgl1JULi-FqGuXlljVXfQBCa3Xg5aSAGm9SslQu1m2I9ITSE6223FgsNjTiBMr4aS-QBeDumyv89n87pAnPpII7wffRZtCW70
Date: Sat, 07 Apr 2018 18:18:55 GMT
Expires: Sun, 07 Apr 2019 18:18:55 GMT
Last-Modified: Sat, 07 Apr 2018 17:25:44 GMT
ETag: "0a8411662813a125f999edad6079d7a5"
x-goog-generation: 1523121944356178
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 41487
Content-Type: application/json
x-goog-hash: crc32c=x7F/0g==
x-goog-hash: md5=CoQRZigToSX5me2tYHnXpQ==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 41487
Server: UploadServer
Cache-Control: no-cache
Age: 57035

curl -I http://storage.googleapis.com/myappname.appspot.com/config/config.json
HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2UpWhRGTwxsME436eKKeSf507oBN4mfT9MZLknOVb92GdRmV-Cs8z8UTVDvoENIeN6fq9PL9A3HkRLgVIWJFhzfVM4Ysy4xz20ZS1Z-Ez18i1islc1o
Date: Wed, 04 Apr 2018 13:34:51 GMT
Expires: Thu, 04 Apr 2019 13:34:51 GMT
Last-Modified: Wed, 04 Apr 2018 07:14:23 GMT
ETag: "522c4e9f5f97873f930501b69da311cf"
x-goog-generation: 1522826063699049
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 40298
Content-Type: application/json
x-goog-hash: crc32c=ZhnEqg==
x-goog-hash: md5=UixOn1+Xhz+TBQG2naMRzw==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 40298
Server: UploadServer
Cache-Control: Cache-Control:no-cache
Age: 333838

It simply refuses to return recent changes that have been made to that file, even after many hours.

UPDATE: TRACEROUTE

traceroute to storage.googleapis.com (216.58.197.80), 30 hops max, 60 byte packets


1  192.168.0.1 (192.168.0.1)  1.544 ms  1.582 ms  1.973 ms
 2  10.247.0.1 (10.247.0.1)  5.637 ms  5.641 ms  5.624 ms
 3  broadband.actcorp.in (202.83.20.173)  5.609 ms  6.521 ms  6.524 ms
 4  broadband.actcorp.in (202.83.20.181)  6.520 ms  6.534 ms  6.941 ms
 5  broadband.actcorp.in (202.83.20.50)  11.394 ms  11.889 ms  11.881 ms
 6  72.14.194.18 (72.14.194.18)  45.704 ms  42.574 ms  42.644 ms
 7  * 108.170.253.113 (108.170.253.113)  42.576 ms *
 8  108.170.237.95 (108.170.237.95)  43.536 ms 108.170.236.197 (108.170.236.197)  45.852 ms  45.865 ms
 9  maa03s21-in-f16.1e100.net (216.58.197.80)  45.847 ms  43.914 ms  45.781 ms

As of today (Apr 10), I still see:

Date: Wed, 04 Apr 2018 15:35:28 GMT
Expires: Thu, 04 Apr 2019 15:35:28 GMT
Last-Modified: Wed, 04 Apr 2018 07:14:23 GMT
ETag: "522c4e9f5f97873f930501b69da311cf"
x-goog-generation: 1522826063699049
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 40298
Content-Type: application/json
x-goog-hash: crc32c=ZhnEqg==
x-goog-hash: md5=UixOn1+Xhz+TBQG2naMRzw==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 40298
Server: UploadServer
Cache-Control: Cache-Control:no-cache
Age: 528505

However, from my office network, I see a newer version of the file, but even there it does not respect the "no-cache" directive and shows stale data occasionally.

Hi, do you retrieve **different** results issuing the same command from the same machine? — GalloCedrone, Apr 08 '18 at 11:19
Its the exact same file and command. That's the weird thing that when I run the curl command one after another, it returns two different responses! — AAP, Apr 08 '18 at 14:05
Yes the file contents also returned are different because we have updated it. But Apr 3 shows older results, Apr 7 the shows newer, but it still does not return the changes we made today (Apr 8)! — AAP, Apr 08 '18 at 14:14
Have you tested this behaviour from different terminals/instances? If you access the object from your browser do you retrieve the updated one? Can you try also with `curl -H 'Cache-Control: no-cache' http://www.example.com`? — GalloCedrone, Apr 08 '18 at 14:28
Different terminal did not make a difference. The browser also shows the older copy, and so does the "Public Link" in Google Storage console. Only tapping the actual file name in Google Storage console shows the latest content (I am also able to retrieve the latest content via gsutil/code) — AAP, Apr 08 '18 at 15:38
curl -H 'Cache-Control:no-cache' http://... also returns the old content — AAP, Apr 08 '18 at 15:40
If you are interested you can open a private Google issue and I will take a look into your project. Register it here issuetracker.google.com/issues/new?component=187164 and post in the comment the link (Disclaimer: I work for Google Cloud Platform Support) — GalloCedrone, Apr 09 '18 at 15:23
Place there (and here) a traceroute to that address to see if you follow always the same path. — GalloCedrone, Apr 09 '18 at 15:24
Thanks Gallo, appreciate your help. I have submitted the issue. Trace route added to original post. — AAP, Apr 10 '18 at 18:24
I am also facing same issue tried many ways. Its bit serious on my side as our finance system is getting lot of issues to get the latest version of the invoices. I hope its solved soon. @GalloCedrone here is the link for it https://issuetracker.google.com/issues/77842189 — Sneha, Apr 12 '18 at 03:17
Sneha, we had to make emergency App updates to pass a dummy parameter to the config ...?nocache=. As of today morning, I am able to receive latest content from the same config, but not sure how long that would be — AAP, Apr 13 '18 at 05:16
@Sneha and APP are you still facing this issue or can it be considered solved also from your side? — GalloCedrone, Apr 20 '18 at 10:07
@GalloCedrone, yes I think it can be considered resolved from my end (though we have anyways updated our Apps now) — AAP, Apr 20 '18 at 13:52
@GalloCedrone yup its solved from my side as well... Thanks a ton !! 1 week nightmare finally ended :D ... — Sneha, Apr 22 '18 at 04:49

GalloCedrone · Accepted Answer · 2018-04-16T08:12:35.153

Disclaimer, I work for Google Cloud Platform Support, I hope you find this information useful.

Currently the engineering team is aware of this issue and they are currently investigating it with high priority.

Note that:

The issue with Google Cloud Storage caching objects that are marked with cache-control: no-cache has been resolved as of 13:11 US/Pacific 2018-04-11. Objects marked with cache-control which may have been cached may be affected until Thursday 2018-04-19, but a workaround is available.

A workaround as been provided:

Customers using Cloud CDN: Customers who have a bucket as a backend for a HTTP(S) Load Balancer may manually expire objects from cache by issuing a cache invalidation through gcloud or the cloud console. Customers not using Cloud CDN: enabling Cloud CDN will cause objects to be served from Cloud CDN going forward which will workaround the issue, as Cloud CDN uses different settings.

Google cloud storage returning older data

1 Answers1