11

I'm facing a little question here. I have a long /etc/hosts file. (working in the web, so I'm hosting a lot of sites locally for developping). Let's says it is about 40, 50 lines long. All redirecting to 127.0.0.1.

Now when checking Chrome's inspector, it seems my local web pages are loading slowly because of the DNS Lookup part. On a mygreatwebsite.local url also defined in the /etc/hosts file.

After a quick check, it would seem that reducing the hosts file to a couple of lines, drastically speeds up the DNS lookup part. So my question, is it possible to be confronted with differences of about 5 seconds for 50 lines, and insignificant times for a couple of lines? Or am I completely missing something?

Zenklys
  • 543
  • 3
  • 5
  • 14

7 Answers7

20

Your tld .local could be the problem, zeroconf uses this. There is no real standard, but the suggested tld for an internal tld is .site or .internal. See http://en.wikipedia.org/wiki/Top-level_domain#Pseudo-domains for this.

ott--
  • 1,091
  • 1
  • 11
  • 13
  • 1
    Indeed, I have noticed great performance boost when using .site extension. After a little research, it seems to be the apple's Bonjour protocol that uses .local tld. – Zenklys Oct 19 '11 at 11:07
  • I joined StackExchange just to say THANK YOU for this excellent tip. My app was taking 10-15 seconds just to load assets and making my dev experience a nightmare. Using .site TLD has the pages loading all snappy again. Awesome. Sidenote: Someone needs to mention this tidbit to all the bloggers/screencasters that use .local! – Steven Garcia Oct 26 '12 at 11:42
6

Can a long /etc/hosts file slow DNS lookup?

The answer is yes, in a Windows (XP/Vista/7) machine.

It is a known problem that a long hosts file does not work well with "DNS Client" service.

http://winhelp2002.mvps.org/hosts.htm:

in most cases a large HOSTS file (over 135 kb) tends to slow down the machine.

[Solution: Set "DNS Client" service to manual or disabled]

http://vlaurie.com/computers2/Articles/hosts.htm:

Note that a hosts file that is much over 100 KB can actually slow up browsing unless the service "DNS Client" is set to manual start.

www.ericphelps.com/scripting/samples/Hosts/:

The most important thing to do before using large HOSTS files is to disable the Domain Name Service client ("DNS Client" or "DnsCache") service on your machine. It sounds bizarre, but the DNS client isn't needed to do DNS. Worse, if you have a large HOSTS file (which is what I suggest), the DNS client chokes your entire PC.

... and I have personally witnessed this on three different Windows machines: XP Pro, Vista, and 7 Starter.

Tero Niemi
  • 161
  • 1
  • Interesting point of view. I didn't knew that. But this questions concerns Mac Os environments. ;) – Zenklys Dec 01 '11 at 08:07
  • This is still relevant with Windows 11; adding a 20MB hosts file (to block every known malicious IP) makes a 24-core i7 /w nvme take 4+ hours to boot. Worse, DNS cache is now system-controlled and cannot be stopped. Do not attempt! – rdtsc Mar 06 '23 at 21:03
2

I would suggest using tcpdump to see whats actually going on. Without it you will be only guessing.

Basic syntax would be 

tcpdump -n -i eth0 port 80 or port 53

(if you are interested in seeing ports 80 and 53, -n to prevent DNS lookups, supposing all comunication is taking place on eth0).

If this is too fast or too much information, redirect it to dumpfile with adding:

 -s 0 -w /tmp/dump

and use i.e. wireshark for looking into the dumpfile. You can see precise timing information in the file and much more...

Radek Hladík
  • 600
  • 1
  • 3
  • 14
1

When i used Spybot it would insert around 50,000 hosts file entries that redirect known malicious sites to 127.0.0.1. I never noticed any issues from having a long hosts file of this nature. Likely has more to do with your dev setup or the code itself. How many sites are you serving locally (50?) Does everyone one of them take this long to respond when 50 entries are present?

iainlbc
  • 2,694
  • 19
  • 19
  • Yes, I'm sure it has nothing to do with the sites itself, they ran perfectly on my other mac (Snow Leopard). Now on Lion, only the server configuration and hosts file have changed. And yes, they all take the same time to load. And always on DNS Lookup part. – Zenklys Oct 19 '11 at 12:29
1

Based on my testing, a 34 MB hosts file with a few million entries slows down browsing in Windows a bit even when the "DNS Client" service is disabled. So, there is a maximum size beyond which you probably don't want to go.

gravitation
  • 111
  • 3
0

This is a slightly tangential answer to your question.

While not in the order of magnitude of seconds, I've observed a measurable difference comparing ping times against hosts when using their IP directly vs. a hostname entered in /etc/hosts mapping to the same IP:

Command Mean [ms] Min [ms] Max [ms] Relative
ping -c 1 XX.XX.XX.XX 13.6 ± 1.3 12.1 32.0 1.00
ping -c 1 <hostname> 14.2 ± 1.0 9.9 26.3 1.05 ± 0.13
ping -c 1 XX.XX.XX.XX 21.2 ± 0.4 18.0 21.9 1.00
ping -c 1 <hostname> 21.8 ± 0.5 20.1 22.9 1.03 ± 0.03
ping -c 1 XX.XX.XX.XX 11.4 ± 0.2 10.0 12.1 1.00
ping -c 1 <hostname> 12.0 ± 0.6 7.2 12.8 1.05 ± 0.06
ping -c 1 XX.XX.XX.XX 7.7 ± 0.3 6.8 9.3 1.00
ping -c 1 <hostname> 8.2 ± 0.4 7.0 9.2 1.06 ± 0.07
ping -c 1 XX.XX.XX.XX 30.6 ± 0.8 25.9 31.4 1.00
ping -c 1 <hostname> 31.2 ± 0.6 26.6 32.0 1.02 ± 0.03
ping -c 1 XX.XX.XX.XX 7.3 ± 0.3 6.1 9.5 1.00
ping -c 1 <hostname> 7.9 ± 0.4 6.8 9.9 1.09 ± 0.08
ping -c 1 XX.XX.XX.XX 8.0 ± 0.3 7.0 9.0 1.00
ping -c 1 <hostname> 8.4 ± 0.4 7.1 9.3 1.06 ± 0.06
ping -c 1 XX.XX.XX.XX 7.3 ± 0.4 6.0 9.1 1.00
ping -c 1 <hostname> 7.7 ± 0.5 6.6 8.6 1.06 ± 0.08

The difference for those hosts is between 2–9% depending on the original ping time.

This is on a mostly out of the box Debian system without any further special configuration related to DNS.

So yes, the lookup takes time, but should not take seconds.

slhck
  • 317
  • 2
  • 17
0

For various reasons I have hosts files on some machines with hundreds of entries. This has no impact on DNS lookups simply because once the machine has found an entry in the hosts file it will not do a lookup for the same information. If everything else is configured correctly educing the size of the hosts file won't affect performance because even with a thousand entries that file can still be parsed faster than performing a DNS lookup. Check the order in which your machine is resolving addresses. The hosts file should be first.

John Gardeniers
  • 27,458
  • 12
  • 55
  • 109