My DNS server is pushing 20mbps, why?

Question

I am running a DNS server in EC2, and it was pushing about 20mbps yesterday when I checked my billing dashboard and found 1.86 TB of used data this month. That's a big bill for my small project lab. I never noticed performance drops and didn't bother to setup traffic threshholds before, but I have now since this has cost me $200+ in bandwidth charges.

It seems someone used my DNS server as part of an amplification attack, however I am at a loss for how.

Config is below.

// BBB.BBB.BBB.BBB = ns2.mydomain.com ip address

options {
        listen-on port 53 { any; };
//      listen-on-v6 port 53 { ::1; };
        directory "/var/named";
        dump-file "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-transfer { BBB.BBB.BBB.BBB; };
        allow-query-cache { BBB.BBB.BBB.BBB; };
        allow-query { any; };
        allow-recursion { none; };

        empty-zones-enable no;
        forwarders { 8.8.8.8; 8.8.4.4; };

        fetch-glue no;
        recursion no;

        dnssec-enable yes;
        dnssec-validation yes;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

zone "mydomain.com" IN {
        type master;
        file "zones/mydomain.com";
        allow-transfer { BBB.BBB.BBB.BBB; localhost; };
};

Given this configuration, I should NOT be answering any queries for zone's I don't host locally right? This server is the SOA for a few domain's, but is not used to look anything up by my other servers (everyone resolves against OpenDNS or Google). What directive do I have wrong here, or am I forgetting? My logs (63MB+) are full of this:

client 58.215.173.155#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 58.215.173.155#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 58.215.173.155#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 58.215.173.155#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 58.215.173.155#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 58.215.173.155#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 218.93.206.228#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 218.93.206.228#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 218.93.206.228#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 218.93.206.228#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 218.93.206.228#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 218.93.206.228#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 50.19.220.154#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 50.19.220.154#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 50.19.220.154#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 50.19.220.154#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 50.19.220.154#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 50.19.220.154#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 123.207.161.124#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 123.207.161.124#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 123.207.161.124#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 123.207.161.124#4444: query (cache) 'cpsc.gov/ANY/IN' denied
client 123.207.161.124#4444: query (cache) 'cpsc.gov/ANY/IN' denied

This doesn't answer your question, but you should set up billing alerts http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html — Tim, Dec 14 '16 at 19:16
Would it be acceptable for you to force fallback to TCP for all clients without RFC 7873 support? — kasperd, Dec 15 '16 at 00:08
@RuiFRibeiro Rate limiting on authoritative DNS servers can be useful. But rate limiting can itself be a weak point which can be exploited in DoS attacks. If an attacker floods a recursor with queries for a domain hosted on an authoritative server with rate limiting, legitimate users of that recursor may no longer be able to resolve records in the domain under attack. That attack can be mitigated with [Aggressive use of NSEC/NSEC3](https://tools.ietf.org/html/draft-ietf-dnsop-nsec-aggressiveuse-07) which is not widely deployed. — kasperd, Dec 15 '16 at 21:09

score 20 · Accepted Answer · answered Dec 14 '16 at 19:57

20

Even if your server is set to only answer authoritative queries as yours is, it's still possible for it to be used for an amplification attack - ANY queries against the root of a zone can trigger a fairly heavy UDP response, since the zone root tends to have a number of records, particularly with SPF/DKIM/DNSSEC.

This is likely what's happening on your system - use tcpdump to confirm. If they are using your authoritative records in an amplification attack, your best options are going to be to simply move to a new IP and hope they don't follow, change your zone root records to make it a less effective amplification vector, or implement response rate limiting (if your BIND supports it).

answered Dec 14 '16 at 19:57

Shane Madden

114,520
13
181
251

They aren't querying my zones though... shouldn't my server be dropping these instead of replying at all? – Russell Anthony Dec 14 '16 at 19:59
4

@RussellAnthony For the log entries you're seeing, yes, I believe it is dropping them - but, for successful attack traffic, no log entry would be created, so in terms of the logs the bandwidth use is invisible. If the attack is still going on (still getting new log entries?) I bet there's a bunch of successful `ANY` queries alongside these failing ones. – Shane Madden Dec 14 '16 at 20:05
2

Added `rate-limit { responses-per-second 1; };` and it seems to have dropped quite a bit of the traffic. I wasn't aware bind could RRL from inside itself. – Russell Anthony Dec 14 '16 at 20:24
@Russell That's what I was going to recommend until I saw your comment. DNS RRL is definitely the right solution for BIND. You can look at enabling the `query-errors` and `rate-limit` logging categories to get more information about the requests you're squashing. – Andrew B Dec 14 '16 at 21:55
@RussellAnthony It probably responds to bogus queries with a very short (< 50 bytes) error code, rather than dropping them on the floor. Still costs money, and serves as a reflection attack, but not an amplification. – Matt Nordhoff Dec 14 '16 at 22:14
1

If they were indeed sending queries for authoritative records it means they must know the domain name. In that case it is not so likely that moving to a new IP will help as they'd be able to find the new IP address as fast as the legitimate users. – kasperd Dec 14 '16 at 22:51
7

Just make sure that your rate-limiting doesn’t turn into a DoS attack vector: if the attacker exhausts the response limit, legitimate caches (such as OpenDNS and google) may fail to resolve your name, too. – Jonas Schäfer Dec 15 '16 at 08:47

My DNS server is pushing 20mbps, why?

1 Answers1