Losing connectivity with DNS

Question

I have set up an 'internal' DNS at my work, basically we have an example.com domain name that is for internet, email etc and I have created on one of our linux network servers (debian) a DNS using bind9 with the domain example.inc.

So based on my files below and the symptoms I'm describing; What can I do to fix this?

These are the critical (I think) files I have modified:

named.conf.local

zone "example.inc" {
        type master;
        file "/etc/bind/zones/example.inc.db";
};
zone "201.168.192.in-addr.arpa" {
        type master;
        file "/etc/bind/zones/rev.201.168.192.in-addr.arpa";
};

named.conf.options

options {
        directory "/var/cache/bind";

        // If there is a firewall between you and nameservers you want
        // to talk to, you may need to fix the firewall to allow multiple
        // ports to talk.  See http://www.kb.cert.org/vuls/id/800113

        // If your ISP provided one or more IP addresses for stable
        // nameservers, you probably want to use them as forwarders.
        // Uncomment the following block, and insert the addresses replacing
        // the all-0's placeholder.

        forwarders {
                1.2.3.4; //IP of our external DNS provider
        };

        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { any; };
};

example.inc.db

$TTL 86400
example.inc.      IN      SOA     ns1.ipower.com. admin.example.inc. (
                                                        2006081401
                                                        28800
                                                        3600
                                                        604800
                                                        38400
)
serv1                IN      A               192.168.201.223
serv2                IN      A               192.168.201.220
serv3         IN      A               192.168.201.219
ns1.ipower.com.      IN      A               1.2.3.4
ns2.ipower.com.      IN      A               1.2.3.5
@                    IN      NS              ns1.ipower.com.
@                    IN      NS              ns2.ipower.com.
svn                  IN      CNAME           serv1
docs                 IN      CNAME           serv2
jira                 IN      CNAME           serv3
confluence           IN      CNAME           serv3
fisheye              IN      CNAME           serv3

rev.201.168.192.in-addr.arpa

$TTL 86400
201.168.192.in-addr.arpa. IN SOA ns1.ipower.com. admin.example.inc. (
                        2006081401;
                        28800;
                        604800;
                        604800;
                        86400
)

223                    IN    PTR    serv1
@                      IN    NS     ns1.ipower.com.
@                      IN    NS     ns2.ipower.com.

named.conf

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";

I then made our internal DNS my preferred DNS with the two external DNSs the next in-line. More the most part this seems to work, I can ping svn.example.inc and it resolves to the correct IP, I can also ping google.com and it also resolves no problem. So all seem good.

However, periodically (couple of times a day at least), I lose the ability to ping the svn.example.inc (and all others defined under the internal DNS). What seem to fix the issue temporarily is to make a change to the network adapter of the client machine and then revert the change. Then it works for a bit but will always fail again.

System Info

Internal DNS

Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.6 (squeeze)
Release:        6.0.6
Codename:       squeeze

Linux 2.6.32-5-686 i686

BIND 9.7.3

PC

OS Name:                   Microsoft Windows 7 Professional
OS Version:                6.1.7601 Service Pack 1 Build 7601
System Type:               x64-based PC

Network Card(s):           2 NIC(s) Installed.
                           [01]: Realtek PCIe GBE Family Controller
                                 Connection Name: WORK LAN
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: the.ipv4.address
                                 [02]: the:ipv6:address

Result of dig +trace

; <<>> DiG 9.3.2 <<>> +trace
;; global options:  printcmd
.                       49341   IN      NS      h.root-servers.net.
.                       49341   IN      NS      k.root-servers.net.
.                       49341   IN      NS      i.root-servers.net.
.                       49341   IN      NS      g.root-servers.net.
.                       49341   IN      NS      a.root-servers.net.
.                       49341   IN      NS      e.root-servers.net.
.                       49341   IN      NS      f.root-servers.net.
.                       49341   IN      NS      d.root-servers.net.
.                       49341   IN      NS      j.root-servers.net.
.                       49341   IN      NS      c.root-servers.net.
.                       49341   IN      NS      b.root-servers.net.
.                       49341   IN      NS      l.root-servers.net.
.                       49341   IN      NS      m.root-servers.net.
;; Received 244 bytes from 192.168.201.223#53(192.168.201.223) in 3 ms

.                       518400  IN      NS      a.root-servers.net.
.                       518400  IN      NS      b.root-servers.net.
.                       518400  IN      NS      c.root-servers.net.
.                       518400  IN      NS      d.root-servers.net.
.                       518400  IN      NS      e.root-servers.net.
.                       518400  IN      NS      f.root-servers.net.
.                       518400  IN      NS      g.root-servers.net.
.                       518400  IN      NS      h.root-servers.net.
.                       518400  IN      NS      i.root-servers.net.
.                       518400  IN      NS      j.root-servers.net.
.                       518400  IN      NS      k.root-servers.net.
.                       518400  IN      NS      l.root-servers.net.
.                       518400  IN      NS      m.root-servers.net.
;; Received 492 bytes from 128.63.2.53#53(h.root-servers.net) in 478 ms

System log during bind9 restart

root@DET4A:~# tail -f /var/log/syslog
Oct 22 14:51:49 DET4A named[17248]: zone 255.in-addr.arpa/IN: loaded serial 1
Oct 22 14:51:49 DET4A named[17248]: /etc/bind/zones/dsasystems.inc.db:12: ignoring out-of-zone data (ns1.ipower.com)
Oct 22 14:51:49 DET4A named[17248]: /etc/bind/zones/dsasystems.inc.db:13: ignoring out-of-zone data (ns2.ipower.com)
Oct 22 14:51:49 DET4A named[17248]: zone example.inc/IN: loaded serial 2006081401
Oct 22 14:51:49 DET4A named[17248]: zone localhost/IN: loaded serial 2
Oct 22 14:51:49 DET4A named[17248]: managed-keys-zone ./IN: loading from master file managed-keys.bind failed: file not found
Oct 22 14:51:49 DET4A named[17248]: managed-keys-zone ./IN: loaded serial 0
Oct 22 14:51:49 DET4A named[17248]: zone example.inc/IN: sending notifies (serial 2006081401)
Oct 22 14:51:49 DET4A named[17248]: zone 201.168.192.in-addr.arpa/IN: sending notifies (serial 2006081401)
Oct 22 14:51:49 DET4A named[17248]: running
Oct 22 14:56:51 DET4A named[17248]: received control channel command 'stop -p'
Oct 22 14:56:51 DET4A named[17248]: shutting down: flushing changes
Oct 22 14:56:51 DET4A named[17248]: stopping command channel on 127.0.0.1#953
Oct 22 14:56:51 DET4A named[17248]: stopping command channel on ::1#953
Oct 22 14:56:51 DET4A named[17248]: no longer listening on ::#53
Oct 22 14:56:51 DET4A named[17248]: no longer listening on 127.0.0.1#53
Oct 22 14:56:51 DET4A named[17248]: no longer listening on 192.168.201.223#53
Oct 22 14:56:51 DET4A named[17248]: exiting
Oct 22 14:56:52 DET4A named[17303]: starting BIND 9.7.3 -u bind
Oct 22 14:56:52 DET4A named[17303]: built with '--prefix=/usr' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc/bind' '--localstatedir=/var' '--enable-threads' '--enable-largefile' '--with-libtool' '--enable-shared' '--enable-static' '--with-openssl=/usr' '--with-gssapi=/usr' '--with-gnu-ld' '--with-dlz-postgres=no' '--with-dlz-mysql=no' '--with-dlz-bdb=yes' '--with-dlz-filesystem=yes' '--with-dlz-ldap=yes' '--with-dlz-stub=yes' '--with-geoip=/usr' '--enable-ipv6' 'CFLAGS=-fno-strict-aliasing -DDIG_SIGCHASE -O2' 'LDFLAGS=' 'CPPFLAGS='
Oct 22 14:56:52 DET4A named[17303]: adjusted limit on open files from 1024 to 1048576
Oct 22 14:56:52 DET4A named[17303]: found 2 CPUs, using 2 worker threads
Oct 22 14:56:52 DET4A named[17303]: using up to 4096 sockets
Oct 22 14:56:52 DET4A named[17303]: loading configuration from '/etc/bind/named.conf'
Oct 22 14:56:52 DET4A named[17303]: reading built-in trusted keys from file '/etc/bind/bind.keys'
Oct 22 14:56:52 DET4A named[17303]: using default UDP/IPv4 port range: [1024, 65535]
Oct 22 14:56:52 DET4A named[17303]: using default UDP/IPv6 port range: [1024, 65535]
Oct 22 14:56:52 DET4A named[17303]: listening on IPv6 interfaces, port 53
Oct 22 14:56:52 DET4A named[17303]: listening on IPv4 interface lo, 127.0.0.1#53
Oct 22 14:56:52 DET4A named[17303]: listening on IPv4 interface eth0, 192.168.201.223#53
Oct 22 14:56:52 DET4A named[17303]: generating session key for dynamic DNS
Oct 22 14:56:52 DET4A named[17303]: set up managed keys zone for view _default, file 'managed-keys.bind'
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 254.169.IN-ADDR.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 2.0.192.IN-ADDR.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 100.51.198.IN-ADDR.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 113.0.203.IN-ADDR.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 255.255.255.255.IN-ADDR.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: D.F.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 8.E.F.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 9.E.F.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: A.E.F.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: B.E.F.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: automatic empty zone: 8.B.D.0.1.0.0.2.IP6.ARPA
Oct 22 14:56:52 DET4A named[17303]: command channel listening on 127.0.0.1#953
Oct 22 14:56:52 DET4A named[17303]: command channel listening on ::1#953
Oct 22 14:56:52 DET4A named[17303]: the working directory is not writable
Oct 22 14:56:52 DET4A named[17303]: zone 0.in-addr.arpa/IN: loaded serial 1
Oct 22 14:56:52 DET4A named[17303]: zone 127.in-addr.arpa/IN: loaded serial 1
Oct 22 14:56:52 DET4A named[17303]: zone 201.168.192.in-addr.arpa/IN: loaded serial 2006081401
Oct 22 14:56:52 DET4A named[17303]: zone 255.in-addr.arpa/IN: loaded serial 1
Oct 22 14:56:52 DET4A named[17303]: /etc/bind/zones/dsasystems.inc.db:12: ignoring out-of-zone data (ns1.ipower.com)
Oct 22 14:56:52 DET4A named[17303]: /etc/bind/zones/dsasystems.inc.db:13: ignoring out-of-zone data (ns2.ipower.com)
Oct 22 14:56:52 DET4A named[17303]: zone dsasystems.inc/IN: loaded serial 2006081401
Oct 22 14:56:52 DET4A named[17303]: zone localhost/IN: loaded serial 2
Oct 22 14:56:52 DET4A named[17303]: managed-keys-zone ./IN: loading from master file managed-keys.bind failed: file not found
Oct 22 14:56:52 DET4A named[17303]: managed-keys-zone ./IN: loaded serial 0
Oct 22 14:56:52 DET4A named[17303]: zone dsasystems.inc/IN: sending notifies (serial 2006081401)
Oct 22 14:56:52 DET4A named[17303]: running
Oct 22 14:56:52 DET4A named[17303]: zone 201.168.192.in-addr.arpa/IN: sending notifies (serial 2006081401)

resolve.conf on DNS

search example.inc
nameserver 209.253.113.18 //This is the IP of the external DNS provider

To be honest, with regards to the resolve.conf file, I am not sure what sort of role it plays on the DNS side.

@Steve-o In order to have the infrastructure be on IPv6 while having the servers themselves still be on v4, for example. Any of v4 and v6 DNS servers can deliver any of v4 and v6 addresses. — glglgl, Oct 18 '12 at 13:32
What do the clients have configured as the DNS resolver IP address? — Sander Steffann, Oct 18 '12 at 13:56
@Steve-o You mean in the sense I keep it checked on my network adapter or the line in the options file **listen-on-v6 { any; };** — Mike Wells, Oct 18 '12 at 13:57
@SanderSteffann The IP address of the server hosting the DNS, in this case 192.168.201.223 — Mike Wells, Oct 18 '12 at 14:07
You mention 'disable IPV6 on the network adapter and the re-enable it'. Which machine is that. The DNS server or the client? — Sander Steffann, Oct 18 '12 at 15:09
That is weird. The client uses IPv4 for DNS resolving and pings over IPv4 as wess (since the hostnsme doesn't have a AAAA record). I have no idea how IPv6 should have any impact on that. Unless the network adapter is buggy and switching IPv6 off/on makes the driver reset something in the adapter that 'accidentally' solves the real problem... — Sander Steffann, Oct 19 '12 at 00:17
Interesting problem... all I can think of is some weird interaction with temporary IPv6 addresses or some general networking problem which gets reset when any network option is changed. Can you still ping the nameserver by its IPv4 address, is IPv6 configured (RA) on the network? — Koos van den Hout, Oct 19 '12 at 20:40
Log entries? Results of `dig +trace`? Anything that might make this answerable? — voretaq7, Oct 19 '12 at 21:05
@voretaq7 - I have added the trace plus a log of when the dns is restarted. Not sure if this this is any help? — Mike Wells, Oct 22 '12 at 19:13
Update - this seems not to be directly linked to IPv6. After running with IPv6 disabled for about an hour I again lost connectivity to the DNS. Re-enabling the DNS 'fixed' the issue. So modifying the network adapter definitely triggers some sort of change. — Mike Wells, Oct 22 '12 at 20:03
How do you diagnose loss of connectivity? What do you have in `/etc/resolv.conf`? Since this is not IPv6 related you may want to update the question. — chutz, Oct 25 '12 at 05:23
@chutz - Currently I am connecting to a jira instance through internal domain name. Periodically I am unable to do this, simply doesn't connect. Then When I ping the domain name it doesn't resolve. The IP does ping though. Further more 'nslookup jira.dsasystems.inc' does return the correct address but 'tracert jira.dsasystems.inc' appears to skip the DNS and go to the internet. When all is working correctly the tracert routes directly to the expected server. — Mike Wells, Oct 25 '12 at 15:08
What about resolv.conf on the client machine? Are you specifying any options? — smithian, Oct 25 '12 at 17:35
@smithian - The client is a windows machine, so I have just added the IP of the DNS as the preferred DNS server. — Mike Wells, Oct 25 '12 at 18:06
This is probably your problem, as Windows does not always use the first DNS server in the list. See this KB article for an explanation and workaround: http://support.microsoft.com/default.aspx?scid=kb;en-us;320760 — smithian, Oct 25 '12 at 18:27
@smithian - thanks for the tip. I have made the reg changes and will let it run for a few days to see if it has fixed the issue. — Mike Wells, Oct 25 '12 at 19:52

score 1 · Accepted Answer · answered Oct 26 '12 at 14:01

Big thanks to smithian for ultimately providing the answer to this one.

The issue seems to be caused by the DNS server priorities not being cleared all the time. It seems that sometimes the preferred DNS is not used and thus the link can't be resolved.

This link on Microsoft Support Site details the issue and provides the solution also.

The Fix

Open registry editor in windows - enter regedit in search window under start menu'=
Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters
Add a new REG_DWORD called ServerPriorityTimeLimit and assign the value 0

This will ensure that the DNS server priorities are reset before deciding what DNS to use.

Losing connectivity with DNS

1 Answers1