0

I have a website with over a million pages served from a varnish cache that sits behind nginx and every day I run a cron to invalidate all the pages (Don’t ask me why, it's complicated…). The website is behind Cloudflare and I am charged per DNS request.

What can I do to reduce the number of URL requests that are sent to Cloudfare? I thought of putting in a separate server prefix that bypasses Cloudflare but realized varnish’s key is the full FQDN+ trailing URL

Quintin Par
  • 4,373
  • 11
  • 49
  • 72
  • Couldn't you just put the website's host in your /etc/hosts file pointing to the direct IP and run the script? –  Aug 23 '17 at 19:01
  • 1
    Disregarding questions on how you ended up down this path, can you elaborate on why there would be such a large number of DNS requests that it would have any notable impact? – Håkan Lindqvist Aug 23 '17 at 19:08
  • @yoonix good point. The web servers are load balanced and the load balancer(s) in different availability zones. Hopefully, I can map the load balancer ips to the domains in hosts – Quintin Par Aug 23 '17 at 19:47
  • @HåkanLindqvist a million*30 days is sizable, right? – Quintin Par Aug 23 '17 at 19:49
  • @QuintinPar But you wouldn't do one DNS query to the authoritative servers per HTTP request? And does "warming the cache" really equal requesting every single page? I would have imagined that there's some significantly smaller subset that constitutes most of the traffic. – Håkan Lindqvist Aug 23 '17 at 19:55
  • @HåkanLindqvist silly me. You've made me think.... I forgot all about DNS being cached. – Quintin Par Aug 24 '17 at 00:25

1 Answers1

0

You could put nginx behind varnish to work as the backend loadbalancer, then tell it to cache the DNS resolutions for longer. That way, varnish will simply be handing off to 127.0.0.1, and nginx can deal with the resolution of the DNS on it's behalf.

Reverend Tim
  • 879
  • 8
  • 14