9

Just noticed from my DNS server's log, showing someone attack my server through port 80:

/var/log/bind.log:31-Jul-2020 03:25:50.536 query-errors: client @0x7f63345948a0 185.107.80.2#36045 (PEACECORPS.GOV): view internet: query failed (REFUSED) for PEACECORPS.GOV/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:31-Jul-2020 05:31:41.446 query-errors: client @0x7f63347273e0 144.217.34.151#53799 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:31-Jul-2020 11:28:20.928 query-errors: client @0x7f63345948a0 2.57.122.193#45066 (.): view internet: query failed (REFUSED) for ./IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:31-Jul-2020 14:21:50.516 query-errors: client @0x7f63345638a0 193.9.17.2#59905 (wzb.eu): view internet: query failed (REFUSED) for wzb.eu/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 07:51:58.756 query-errors: client @0x7f6334718db0 89.248.168.17#37241 (cpsc.gov): view internet: query failed (REFUSED) for cpsc.gov/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 18:09:37.112 query-errors: client @0x7f633801db20 83.97.20.164#21544 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:03.982 query-errors: client @0x7f6334689490 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:04.263 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:04.333 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:04.708 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:22.091 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:22.534 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:23.634 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:12:26.022 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:01.519 query-errors: client @0x7f63347d8eb0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:02.432 query-errors: client @0x7f63346f2650 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:19.174 query-errors: client @0x7f63345948a0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:20.556 query-errors: client @0x7f633801db20 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:35.657 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:39.615 query-errors: client @0x7f633c0da830 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:51.414 query-errors: client @0x7f63345948a0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:13:57.623 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:14:07.363 query-errors: client @0x7f63346f2650 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:14:15.991 query-errors: client @0x7f6334771730 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:14:25.212 query-errors: client @0x7f63347ca880 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:14:32.046 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:14:43.583 query-errors: client @0x7f6334775120 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:14:50.684 query-errors: client @0x7f6338289e50 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:01.011 query-errors: client @0x7f633c0da830 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:08.899 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:17.051 query-errors: client @0x7f63347e74e0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:26.382 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:33.001 query-errors: client @0x7f63347ca880 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:44.613 query-errors: client @0x7f63346f2650 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:15:49.267 query-errors: client @0x7f63345948a0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:02.472 query-errors: client @0x7f6334775120 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:04.881 query-errors: client @0x7f6338289e50 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:20.139 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:21.184 query-errors: client @0x7f6334718db0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:37.295 query-errors: client @0x7f63346f2650 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:37.725 query-errors: client @0x7f63346f2650 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:53.255 query-errors: client @0x7f6334775120 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:16:55.799 query-errors: client @0x7f6334775120 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:09.169 query-errors: client @0x7f63346f2650 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:14.215 query-errors: client @0x7f6334771730 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:25.206 query-errors: client @0x7f63347d8eb0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:31.728 query-errors: client @0x7f633827b820 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:40.997 query-errors: client @0x7f63381611b0 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:50.548 query-errors: client @0x7f633827b820 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145
/var/log/bind.log:01-Aug-2020 19:17:57.181 query-errors: client @0x7f63381dba30 37.49.224.64#80 (sl): view internet: query failed (REFUSED) for sl/IN/ANY at /bin/named/query.c:7145

Wonder how could they be able to do it?

My DNS server only allow port 53 and 22 to get in, and also have processes to monitor and block these kinds of IP via firewall-cmd command like this:

firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="37.49.224.64" drop' --permanent

This usually works for most of attacks, but didn't work for this IP, tried to change zones from public to block, and action from drop to reject, nothing works.

End up I have to block this IP from external firewall, then finally get it out.

Wonder does any one saw these kinds of attacks? And how can they bypassing local Firewalld?

Any suggestion will be appreciated!

watchsat
  • 101
  • 1
  • 5
  • 7
    The port logged is the **source port**, not the destination port on your server. The IP address [`37.49.224.64`](https://www.abuseipdb.com/check/37.49.224.64) does have a bad reputation. – Esa Jokinen Aug 02 '20 at 04:46
  • While they are hitting the server, that seems like a pretty slow rate of attack. I may be reading it wrong but it looks like whole seconds go by between requests. – Mark Rogers Aug 03 '20 at 02:18

2 Answers2

19

You're looking at the client query logs, and normally a client will choose from one of the ephemeral ports to have your DNS server respond back to. Yes your server is listening on port 53, but your clients will most likely receive responses from your DNS server over ports 49152 to 65535. The fact that the source of your query traffic is choosing to use port 80 as the return trip port is... odd, but virtually irrelevant. I'm sure it's some method of circumventing network security on the client's side. Or the developers of whatever software is attempting to abuse your DNS server were just not particularly concerned with using ephemeral ports. Who knows.

As for your firewall, you need to run either firewalld-cmd --reload or firewalld-cmd --complete-reload afterwards to make sure that the rule is processed.

EDIT:

This IP can obtain local port 80 every time get in.

To be clear, port 80 in your logs is not referring to your DNS server at all. That is purely referring to the return trip that packets will take to get back to the client. When you see this:

37.49.224.64#80

That means DNS responses will be returned to 37.49.224.64:80, just like when you see this in the first line of your logs:

185.107.80.2#36045

Any DNS query that your DNS server satisfied was returned to 185.107.80.2:36045

To reiterate: No traffic is coming to your server over port 80, just like no traffic is coming to your server over port 36045. Those return trip ports are completely, utterly, and absolutely irrelevant to you.

This is, at its heart, a firewall misconfiguration. Either through firewalld zones, interfaces, rule ordering, or reloading issues.

Wesley
  • 32,690
  • 9
  • 82
  • 117
  • Yes, usually client side won't use any port below 1000, and that never happened in the past. This IP can obtain local port 80 every time get in. Also did the reload, even restart firewalld demon but no help... – watchsat Aug 02 '20 at 04:42
  • @watchsat Updated my answer – Wesley Aug 02 '20 at 04:48
  • 1
    Ha, you are right... For some reason I keep thinking that's destination port, and I'll check the setting for my firewall. Many thinks for the help! – watchsat Aug 02 '20 at 05:00
  • 1
    @watchsat It's all part of troubleshooting in general - whittling down possibilities, and clearing up ambiguities. I think you can now focus on firewall topics from here on out as you work through it. It may be useful to make a new question that focuses purely on the networking aspect. _"I have Rule A to block Traffic B, but Traffic B is still getting through. How can I troubleshoot this?"_ You'll probably get good responses as to where to look next. – Wesley Aug 02 '20 at 05:03
  • 1
    I think I found out the issue... After removed NetworkManager and change back to eth0, actually the firewalld was running but somehow not working. Ran the command "firewall-cmd --get-active-zones" got empty result, then after add-interface and restart demon now it looks fine. – watchsat Aug 02 '20 at 05:46
  • Now I have to wait for next run of attack to make sure it works... haha... – watchsat Aug 02 '20 at 05:48
  • @watchsat Great! Sounds like you’re on the right track. – Wesley Aug 02 '20 at 07:11
  • 5
    Sounds like a reflector attack - someone sending a shit-ton of queries with a forged origin IP and port, causing your nameserver to send packets to that IP/port in response. – Shadur Aug 02 '20 at 12:42
6

The traffic is returning to the external IP's port 80. Usually, the provided “source IP address” is spoofed, and somebody's trying to use your DNS server as part of a DDOS attack – specifically, a reflector attack.

You should set up something like fail2ban to prevent your server from being used in such an attack. Or, to mitigate it, just configure your firewall to reject client ports below 1024.

wizzwizz4
  • 171
  • 7
  • Yes, I wrote a perl script to do similar job like that; as wizzwizz4 mentioned, my firewalld didn't config correctly. Guessing because I was using iptables in the past and assumed that firewalld is same as that, didn't setup interface for the zone I was using. Now everything looks good. Thanks! – watchsat Aug 02 '20 at 14:37
  • How's fail2ban going to help with stateless UDP nonsense? – joshudson Aug 03 '20 at 15:34
  • @joshudson By dropping the incoming UDP nonsense. – Michael Hampton Aug 03 '20 at 15:42
  • @MichaelHampton: Where's the fail part? – joshudson Aug 03 '20 at 15:43
  • 1
    @joshudson The log entries in the OP are easily parseable for this purpose. – Michael Hampton Aug 03 '20 at 15:43
  • I'm using Perl to get the external IPs like this: @extiplive = `cat /var/log/bind.log | grep REFUSED | cut -d' ' -f6 | awk -F\# '{print $1}' | sort | uniq` ; Then read the IPs which already blocked like this: @existip = `grep address /etc/firewalld/zones/public.xml | awk -F\" '{print $2}'` ; Compare both arraies and block new IPs by this command: firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="$NEWIP" reject' --permanent Run this script every 15 seconds on my DNS server, it will block everything from those IPs, no matter UDP or TCP... – watchsat Aug 04 '20 at 03:12
  • The fail part was didn't setup interface to public zone correctly; before I fixed that some of the IPs (like this IP 37.49224.64) just got ignore and kept coming. After run command: firewal-cmd --permanent --zone=public --add-interface=eth0 all the issues were gone. – watchsat Aug 04 '20 at 03:19
  • @watchsat Ooh, don't block them permanently, if this is a public service. Then somebody could DOS your users forever by spoofing *their* IP addresses. – wizzwizz4 Aug 04 '20 at 09:08
  • no worry, this method also can be used on other web and mail services, too. Once collect them and store into main database, you can generate IP block to ban them all. – watchsat Aug 04 '20 at 17:34
  • @watchsat No, I mean… let's say that user A had the IP address 54.253.7.12. If a bad actor sent loads of requests with a return address 54.253.7.12, your security system would ban user A forever. – wizzwizz4 Aug 04 '20 at 18:09
  • Oh, that won't be possible in my case, only "hacking" kinds of activities I will collect and block them. :) – watchsat Aug 04 '20 at 18:30
  • @watchsat But you're not necessarily blocking the actual source of those activities; just the "return to" address. There's no reason to expect that the attacker isn't just _lying_ about that address; in fact, the attack described in the question is predicted on that behaviour! (Some parts of the internet filter out packets that are _known_ to be lying about the return IP address, but there's always a way to make the packets have plausible deniability.) – wizzwizz4 Aug 05 '20 at 15:53
  • Yes, the "return to" can be ignored for those spam attacks. Used to work for large ISP and built anti-spam system long time ago, only target source IPs. – watchsat Aug 06 '20 at 16:23
  • @watchsat How do you do that? I was under the impression that the source IP was easily spoofable. – wizzwizz4 Aug 06 '20 at 17:02
  • If I remembered correct, I was checking each mail's header and get the source IP. – watchsat Aug 06 '20 at 18:15
  • @watchsat Unless the protocol involves a three-way handshake (e.g. TCP), the source IP should not be trusted in the slightest. So, for UDP it can't be trusted to be correct; you can spoof it fairly easily. – wizzwizz4 Aug 07 '20 at 12:09