2

I have set up nginx reverse proxy and added caching to it using nginx caching.

I can verify whether the requested copy is cached or not using added header.

add_header X-Cache-Status $upstream_cache_status;

i have set to cache page if its get accessed more than twice.

proxy_cache_min_uses    2;

Nginx caching is working as it should, i can see the X-Cache-Status: MISS twice and later X-Cache-Status: HIT if i try to access the webpage from any browser.

But if i try to check same with curl , if the page is not cached i am getting 

X-Cache-Status: MISS

and if the page is cached i am getting 

X-Cache-Status: HIT

But my issue is , i cant able to warm up cache using curl. i mean i cant make webserver make cache specific page by accessing page using curl even if i access it 10 times using curl.

why is that ?

So far i have tried .

curl -I https://example.com/page1
curl -v https://example.com/page1
curl --verbose https://example.com/page1
curl -svo /dev/null https://example.com/page1

using latest centos 7x64

latest Nginx

[root]# curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

update1: here is my nginx config file

proxy_cache_path    /usr/local/nginx/cache levels=1:2   keys_zone=mycache:10m       max_size=10g    use_temp_path=off;

server {

access_log  /dev/null;
listen my_ip:80;
server_name example.com ;

location / {
    access_log off;

    proxy_cache             mycache;
    proxy_cache_min_uses    2;
    proxy_cache_valid       200     10m;
    proxy_cache_valid       404     1m;

    proxy_cache_use_stale   error timeout invalid_header updating http_500 http_502 http_503 http_504;  
    proxy_cache_lock        on;

    proxy_pass http://box1.example.com/;

    proxy_set_header    X-Real-IP   $remote_addr;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;

    #hide headers 
    proxy_hide_header Cache-Control;

    #ignore header
    proxy_ignore_headers Cache-Control Expires;

    #add custom headers for users
    add_header X-Cache-Status $upstream_cache_status;
    expires 3h;
    add_header Cache-Control "public, max-age=10800, s-maxage=10800";

    #add no mobile redirect cookie
    add_header Set-Cookie "__cf_mob_redir=0;Domain=.example.com;";
}
}

update 2:
I am using cloudflare on my domain. I have tried disabling cloudflare service altogether and only using it as a DNS server . Tried again with curl and got same response.

It does not matter whether cloudflare is enabled or not curl cant make pages cache.

Update 3 :

Tried using different User-agnts but received same response.

curl -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" -svo /dev/null http://example.com/some-page
  • Please share your Nginx config, especially the part around caching. – Tim Nov 17 '16 at 04:28
  • You're using old http1 cache control headers, have a read of a tutorial I wrote about that https://www.photographerstechsupport.com/tutorials/hosting-wordpress-on-aws-tutorial-part-4-wordpress-website-optimization/#ccheaders . This doesn't answer your question but is worth considering. Short version: only header required is "Cache-Control". Unfortunately I can't answer your curl question. If it's not causing a problem I'd probably let it go, but it'd be interesting to get an answer. – Tim Nov 17 '16 at 18:47
  • @Tim Thanks updated config file. and still the same. –  Nov 23 '16 at 13:49
  • Do you see cache misses from web browsers? Try Firefox and maybe Chrome with the plug "Live HTTP Headers" it's ideal for this. If web browsers work but curl doesn't then there's no actual problem, just curiosity. – Tim Nov 23 '16 at 18:22
  • @Tim yes, i can see MISS twice then HIT afterwards in browser. practically its working perfectly. but i would like to warm up cache from time to time. so i need curl to cache pages too. and i am using cloudflare on my website, i even tried disabling cloudflare and trying with curl, but still the same. –  Nov 25 '16 at 05:25
  • Ah hold now now, you've just given us really significant new information, that you're using CloudFlare. Perhaps you should edit your question to give us the full story, even the parts you don't think are relevant. – Tim Nov 25 '16 at 06:29
  • @Tim updated question. –  Nov 25 '16 at 06:39
  • I just tried this on my website. I use php and fastcgi_cache, not proxy cache. Either using CloudFlare public IP or server private IP I get the cached page using curl or Firefox. Nothing in your config stands out as obviously incorrect. I'd try paring it back to the absolute minimum. After that try adding proxy_cache_methods, but it should default to allow GET anyway. Do some thinking around user agent affect on caching. I could think some more but since this isn't actually causing a problem I see no need. – Tim Nov 25 '16 at 20:41
  • try setting the accept-encoding header. Post the send and received headers of the third run of this, leaving around 10 seconds between runs: curl -v --header "Accept-Encoding: gzip, deflate" https://www.yourwebsite.com > out.txt – Tim Nov 26 '16 at 03:18
  • @Tim i tried curl -v --header "Accept-Encoding: gzip, deflate" yourwebsite.com > out.txt but out.txt file is garbled or non readable text. –  Nov 26 '16 at 12:56
  • on Centos / Amazon Linux the send and receive headers all came out, including the cache status. Change the command however is required in order to see the headers sent back by the web server - try removing >out.txt first I guess – Tim Nov 26 '16 at 18:58

1 Answers1

1

0) As I can see you check https:// site but configuration for http:// version. I guess it's not a problem in your case but better to check that you edit right configuration file.
1) You should check what you got from upstream curl -I http://box1.example.com/page1 and I guess that there will be Set-Cookie header. So you should either add Set-Cookie to proxy_ignore_headers or remove cookies from upstream's reply. Or use separate locations: one for pages where you could ignore cookies for caching and another for pages where you can't ignore cookies.

Fedor Dikarev
  • 736
  • 4
  • 10