2

Just started using google cloud storage. I am having problem receiving gzip files from it. Stored file is gzip'ed but it gets decompressed on request. I have read all documentation, forums and have tried dozens of different combinations of headers and still fetched file in browser is not compressed. Could some one point out what I am doing wrong?

Article about gzip, headers, transcoding https://cloud.google.com/storage/docs/transcoding

I move my file to storage with this command 

gsutil -h "Cache-Control:public,max-age=10,no-transform" cp  -Z bundle.js gs://lol-champs.tomdid.com

File gets gziped before uploading file and size decreases by 5 times once it gets on storage, I have checked in console.

Browser request headers

Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8,lt;q=0.6
Cache-Control:no-cache
Connection:keep-alive
Host:lol-champs.tomdid.com
Pragma:no-cache
Referer:http://lol-champs.tomdid.com/

Response headers

Accept-Ranges:bytes
Access-Control-Allow-Origin:*
Access-Control-Expose-Headers:Content-Encoding
Access-Control-Expose-Headers:Content-Length
Access-Control-Expose-Headers:Content-Type
Cache-Control:public,max-age=10,no-transform
Content-Language:en
Content-Type:application/javascript
Date:Sat, 15 Oct 2016 21:40:14 GMT
ETag:"8505f976c2ffbf1e69c3ee9fdf2f04d5"
Expires:Sat, 15 Oct 2016 21:40:24 GMT
Last-Modified:Sat, 15 Oct 2016 21:40:09 GMT
Server:UploadServer
Transfer-Encoding:chunked
x-goog-generation:1476567609681000
x-goog-hash:crc32c=Gik6aA==
x-goog-hash:md5=hQX5dsL/vx5pw+6f3y8E1Q==
x-goog-metageneration:1
x-goog-storage-class:STANDARD
x-goog-stored-content-encoding:gzip
x-goog-stored-content-length:290207

So google sets custom header that content is gziped in storage but it sends back response to me decompressed and without Content-Encoding header.

Not sure is the way how I upload file is wrong, or it is missing some headers. feeling lost. Link to File -> http://lolchamps.tomdid.com/bundle.js

Any help would be much appreciated.

UPDATE: Found something interesting, if I am requesting file with googles URL- https://storage.googleapis.com/lolchamps.tomdid.com/bundle.js it returns gziped file, if the same file is requested via my domain it returns decompressed version. WHY??? I smell dark magic.

Oozhaa
  • 181
  • 1
  • 12
  • Can I ask you sth. When you upload the file, the file is compressed at google cloud storage servers or at client side?Is there such option? – curious May 09 '17 at 16:59
  • @curious it is compressed using gsutils when uploading. Check https://cloud.google.com/storage/docs/gsutil/commands/cp -Z option – Oozhaa May 10 '17 at 20:43
  • I have the exact same problem here with 2 behaviors depending of the URL as mentioned in your update. This is dark magic indeed. Did you solve this? – probitaille Sep 01 '20 at 21:21

2 Answers2

0

I can't seem to reproduce this...?

$ curl -v http://lolchamps.tomdid.com/bundle.js
[...]
< Content-Length: 290667
< Content-Encoding: gzip
[...]

Edit: Same with Chrome, full request as follows:

GET /bundle.js HTTP/1.1
Host: lolchamps.tomdid.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36
Accept: */*
DNT: 1
Referer: http://lolchamps.tomdid.com/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,de;q=0.6,de-DE;q=0.4

And the response:

HTTP/1.1 200 OK
Date: Tue, 18 Oct 2016 17:27:43 GMT
Cache-Control: no-transform
Expires: Wed, 18 Oct 2017 17:27:43 GMT
Last-Modified: Sun, 16 Oct 2016 22:53:39 GMT
ETag: "7c79e6fd21a30bf99f0344a7fb9f6105"
x-goog-generation: 1476658419217154
x-goog-metageneration: 1
x-goog-stored-content-encoding: gzip
x-goog-stored-content-length: 290667
Content-Type: application/javascript
Content-Encoding: gzip
Content-Language: en
x-goog-hash: crc32c=vZKmgQ==
x-goog-hash: md5=fHnm/SGjC/mfA0Sn+59hBQ==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 290667
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Type
Access-Control-Expose-Headers: Content-Length
Access-Control-Expose-Headers: Content-Encoding
Server: UploadServer
lot
  • 1,045
  • 1
  • 8
  • 8
  • Thank you for checking this out. That is right curl gives right response, with right headers. However chrome in developer tools (Network tab) shows that file is fetched not gziped( file size 1.1mb and there is no Content-Encoding header). Any ideas why chrome and curl gives different results? Thanks. – Oozhaa Oct 17 '16 at 19:21
  • I just tried it in Chrome and I get gzip encoded content as well, just like curl. My browser's user agent is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36 – lot Oct 17 '16 at 21:42
  • Hm, the responses I'm getting have a different cache control header: "Cache-Control: no-transform" vs "Cache-Control:public,max-age=10,no-transform" that you cite. – lot Oct 17 '16 at 21:46
  • This is so strange. I just checked mozilla and safari, both of them returns decompressed versions of bundle.js. Could you post your full request and response headers? Thanks. – Oozhaa Oct 18 '16 at 07:56
  • Is it possible that your requests are going through some intermediate proxy what might fudge with the encoding? – lot Oct 18 '16 at 17:18
0

When your clients uses Accept header to signal that it supports content(-encoding) compression (e.g., gzip) then GCP returns compressed response (because your client asked for it) but does not include Content-Length header. There is X-Goog-Stored-Content-Length header instead, which tells the size of original contents.

Curl does not ask for compressed response by default and this is why you do not see the issue there. You can use curl --compressed to ask for compressed response and if you do curl --compressed -D - you will see no Content-Length.

Mitar
  • 6,756
  • 5
  • 54
  • 86