0

We're having a bizarre issue with one of our domains that has been going on for the past 15+ hours at this point. The domain is not resolvable in many parts of the country. I tried pinging it on a few different Linux servers and got:

example.com: Temporary failure in name resolution

The problem isn't geographically limited though. People all over the country have now told me they can't ping the domain either. However, some people can ping the domain and access it normally.

Finally, somebody pointed me to this site: https://dnschecker.org/ip-blacklist-checker.php

When I try resolving the domain, "Blacklisted?" is No for all of them except for this one: dnsbl.spfbl.net

When I click the error, it says this:

No rDNS was found.

This IP has been flagged because have none valid FCrDNS.

Register a valid rDNS for this IP, which points to the same IP.

The rDNS must be registered under your own domain for you be able to delist it.

I can't think what has caused this or how to fix it. The domain's A records don't point to a static IP as far as I know, since it goes through the Cloudflare load balancers. I thought it might have been an issue with Cloudflare, which we use for our nameservers. However, their portal didn't indicate anything at all yesterday or anything relevant this morning.

The only fix we've found is to change the preferred DNS server manually. For instance, Google and OpenDNS don't resolve the domain. On dnsblacklist.org, 7 out of 12 DNS resolvers fail to resolve it. Cloudflare and 4 other resolvers can successfully resolve it. If we change the preferred DNS server to 1.1.1.1 manually on the servers, now they can ping the domain.

The problem is that all of our public users, obviously, are not going to do this. Any thoughts on what exactly is going on? I haven't changed anything with the domain records at all recently, and this doesn't affect any of my other domains which also use Cloudflare as the nameserver. The issue affects all subdomains on this domain as well as the primary domain, regardless of whether traffic on that address is proxied through Cloudflare.

UPDATE:

Another domain:

enter image description here

Problematic domain:

enter image description here

InterLinked
  • 146
  • 1
  • 8
  • What is the domain name? – Michael Hampton Oct 19 '20 at 19:19
  • 1
    "The domain is not resolvable in many parts of the country. I tried pinging it" ping is never the appropriate tool to troubleshoot DNS problems. Do DNS queries using `dig` first towards authoritative nameserver then towards recursive ones from multiple vantage points and provide all of that if you want anyone to be able to help. Otherwise use the online checking tools like DNSViz or Zonemaster on your domain as first step. You might also want to ask your DNS provider. – Patrick Mevzek Oct 19 '20 at 22:54
  • @PatrickMevzek I have checked the DNS records from multiple online DNS checkers. They all say that around half of their servers can't resolve the domain – InterLinked Oct 19 '20 at 23:00
  • There is certainly a reason but without knowing the domain name it is like impossible to pinpoint the real one... You are not even specifying which services you use to test. For example do they check DNSSEC? Did you check domain status in whois (is is on hold? expired?)? Do you use glues? etc. Without knowing the domain it is impossible to help you. The DNS is public, what do you gain exactly by refusing to disclose it? – Patrick Mevzek Oct 19 '20 at 23:08
  • @PatrickMevzek The domain is phreaknet.org – InterLinked Oct 19 '20 at 23:20
  • `phreaknet.org` lists 172.67.175.181 as one of the DNS Servers for this domain, which is not responding to any query (at least not from UK, FR an NL). Remove that machine from your zone. – bjoster Oct 28 '20 at 10:58

1 Answers1

2

If you go to https://dnsviz.net/d/phreaknet.org/X44olg/dnssec/ you will find, right now, no less than 6 bogus errors and 3 errors on your domain. Said differently: it is horribly broken at the DNS layer:

DNSViz report for phreaknet.org on 2020-10-20 00:00:22 UTC

I don't know which services you used to check, but obviously they were oblivious to DNSSEC problems, hence they are not good. Use dnsviz.net, it is tried and trusted.

As said in comments, DNSSEC is often a source of "it works there but not there" because if it is broken it will be seen as broken only by validating resolvers, which are not all resolvers out there. Plus there are many edge cases in DNSSEC that can make one resolver accept the answer and not the other.

You will need to fix all of these problems.

Start by completely removing DNSSEC off it, unless you are sure to master it. Which means go to its currently sponsoring registrar (Namecheap as seen in whois) and find the way to remove DS records at the registry for your domain.

Once this is done you will need to wait. How much?

.ORG authoritative nameservers publish your DS record with a TTL of one day:

$ dig org. NS +short
b2.org.afilias-nst.org.
b0.org.afilias-nst.org.
a2.org.afilias-nst.info.
a0.org.afilias-nst.info.
d0.org.afilias-nst.org.
c0.org.afilias-nst.info.
$ dig @d0.org.afilias-nst.org. DS phreaknet.org +noall +ans
phreaknet.org.      1d IN DS 2371 13 2 (
                1AE4B3ECD5282A0A412E6792B60855C39D9166F94703
                980CCECB740EAAD501A9 )

So you need to wait for at least one day after the moment where this record disappears (which can happen some time after you ask your registrar to remove it).

Once the DS record has disappeared for at least 1 day, then do a DNSviz check again and see if you have other problems. But it is probably with just that you will have solved all your problems already.

Alternatively, you should ask your DNS provider to help you for DNS related problems on your domain. Especially since the core problem is: "RRSIG phreaknet.org/DNSKEY alg 13, id 2371: The Signature Expiration field of the RRSIG RR (2020-10-18 21:55:18+00:00) is 1 day in the past." which means an error by your DNS provider not updating signatures on your domain, or by yourself if you were instructed to change the DS record at registry to use another key by your DNS provider and you didn't do it. Expiration of this signature recently explains why you started to get reports of problems around one day ago.

If your DNS provider can regenerate valid signatures, with the current key, on all records, since things will again work without having to remove the DS at registry. But you will also need to wait, albeit far less as signatures' TTLs seems to be either 1 hour or 5 minutes:

$ dig phreaknet.org SOA +dnssec +cd +noall +ans
phreaknet.org.      1h IN SOA donna.ns.cloudflare.com. dns.cloudflare.com. (
                2035213272 ; serial
                10000      ; refresh (2 hours 46 minutes 40 seconds)
                2400       ; retry (40 minutes)
                604800     ; expire (1 week)
                3600       ; minimum (1 hour)
                )
phreaknet.org.      1h IN RRSIG SOA 13 2 3600 (
                20201021011353 20201018231353 34505 phreaknet.org.
                bxVvIlM7M5tlKDLsRUSj2LSpNrvkv4DlZnzi+AfKFkP7
                1GuzfyOJmNlpQK+eD8j+kigLcBXUEkbO66rY76QWtA== )
$ dig phreaknet.org A +dnssec +cd +noall +ans
phreaknet.org.      5m IN A 104.18.57.190
phreaknet.org.      5m IN A 104.18.56.190
phreaknet.org.      5m IN A 172.67.175.181
phreaknet.org.      5m IN RRSIG A 13 2 300 (
                20201021011314 20201018231314 34505 phreaknet.org.
                BCZjG6JZfJqERRBOZ9DP50OBXzUhCdQ777EoElWv52oU
                xRRWw+I1e/ok3a5V2h1gt/OBLVVwlRFQhALk2rclSA== )

Aside useful note about the above:

  • you need +dnssec to see the RRSIG records and hence their TTL (otherwise they are not shown by default because never useful per se... except to troubleshoot DNSSEC related problems)
  • you absolutely need +cd which is the flag that requests NOT to do DNSSEC validations, otherwise those queries will probably just get back SERVFAIL because the DNSSEC configure is broken. +cd is the most important flag of dig for DNSSEC: "do a query without it and get SERVFAIL then redo same query with it and get a reply" means in 99.999% of cases that your problem is solely related to DNSSEC configuration on your domain.
Sirex
  • 5,499
  • 2
  • 33
  • 54
Patrick Mevzek
  • 9,921
  • 7
  • 32
  • 43
  • This is interesting, thanks! I'm flummoxed because I have DNSSEC on all my domains and this is the only one with this issue. Any ideas why that might be? – InterLinked Oct 20 '20 at 00:16
  • See the end of answer: signatures have expired ~26 hours ago. Which is a sign of signatures renewal broken at DNS provider or key rotation process failure (if you needed to upload a new DS record at registry, based on your DNS provider instructions). – Patrick Mevzek Oct 20 '20 at 00:17
  • "I have DNSSEC on all my domains" Make sure to add **DNS monitoring** on all your domains, and especially for DNSSEC (like for X509 certificates) make sure to monitor the signatures (on some given records) expiration dates (and inception dates as well). – Patrick Mevzek Oct 20 '20 at 00:19
  • I looked in Cloudflare at DNSSEC and I screenshot what I saw for two different domains. There is definitely a difference. I will check the registrar and see what's there. Based on that, this seems to indicate a problem with Cloudflare, wouldn't you say? – InterLinked Oct 20 '20 at 00:29
  • The signatures has expired. Your DNS provider is in charge normally of the DNSSEC configuration of your domain. However many other thing may have happened where he is not the culprit: maybe he told you it won't do DNSSEC on the domain anymore and asked you to remove the DS records (but you didn't). Or it asked you to change the DS records at the registry to use another key (but you didn't). Or many other things. That is impossible to say, you need to ask it. The DNS just shows that signatures are expired. – Patrick Mevzek Oct 20 '20 at 01:59
  • OK, so it seems that simply toggling DNSSEC off and on did the trick. When I toggled it back on, I looked for new DS info but it was all the same. I didn't change anything in the domain registrar at all. Somehow though, issues went away. Reran the test you linked and it's now showing success, and I can now resolve the site myself. I think somehow this was a glitch with Cloudflare and toggling it off/on took care of it. – InterLinked Oct 20 '20 at 02:04
  • Good if you found a quick solution then. But not knowing the real root cause means it can happen anytime again. I would strongly suggest you ask your provider for explanations and separately that you add monitoring on your domains to make sure DNSSEC (and DNS) works continuously, and to get alerts because your clients/users/customers alert you. – Patrick Mevzek Oct 20 '20 at 02:10