We are using fail2iban on Ubuntu 18 to detect bad login attempts and ban the IP addresses which try too many times. We use iptables for the IP blocking.
After using it for a while we noticed that some of the bans do not work - some IP addresses are banned properly while some are allowed to make more login attempts and nothing seems to stop them.
Did anybody experience strange issues with iptables like his? I see many tickets about iptables not working at all, but our issue is different - wast majority of the blocking works and only some small amount of it fails.
The issue is not with fail2ban. We log iptables -vnL every minute to make sure the IP addresses do not disappear from the list for no reason.
A.B. made a great suggestion that because we use iptables with REJECT the attacker might be using an already established connection to send in more bad requests. So we adjusted the rules to use DROP.
But the issue is still not fixed. We noticed bad requests coming from 2.58.149.35 which was on the list of at least a couple of days.
This is how the rules in iptables look like (iptables -vnL):
Chain INPUT (policy DROP 66 packets, 3358 bytes)
pkts bytes target prot opt in out source destination
37M 131G f2b-fv-dos tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 0:65535
14M 1466M f2b-wordpress tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
14M 1467M f2b-waf tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
14M 1476M f2b-repeated tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
238M 821G ufw-before-logging-input all -- * * 0.0.0.0/0 0.0.0.0/0
238M 821G ufw-before-input all -- * * 0.0.0.0/0 0.0.0.0/0
201K 657M ufw-after-input all -- * * 0.0.0.0/0 0.0.0.0/0
199K 657M ufw-after-logging-input all -- * * 0.0.0.0/0 0.0.0.0/0
199K 657M ufw-reject-input all -- * * 0.0.0.0/0 0.0.0.0/0
199K 657M ufw-track-input all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ufw-before-logging-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-before-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-after-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-after-logging-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-reject-forward all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ufw-track-forward all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 47 packets, 4596 bytes)
pkts bytes target prot opt in out source destination
237M 1038G ufw-before-logging-output all -- * * 0.0.0.0/0 0.0.0.0/0
237M 1038G ufw-before-output all -- * * 0.0.0.0/0 0.0.0.0/0
845K 798M ufw-after-output all -- * * 0.0.0.0/0 0.0.0.0/0
845K 798M ufw-after-logging-output all -- * * 0.0.0.0/0 0.0.0.0/0
845K 798M ufw-reject-output all -- * * 0.0.0.0/0 0.0.0.0/0
845K 798M ufw-track-output all -- * * 0.0.0.0/0 0.0.0.0/0
Chain f2b-fv-dos (1 references)
pkts bytes target prot opt in out source destination
37M 131G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain f2b-repeated (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 187.54.60.43 0.0.0.0/0
14M 1475M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain f2b-waf (1 references)
pkts bytes target prot opt in out source destination
30 1520 DROP all -- * * 2.58.149.35 0.0.0.0/0
14M 1467M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain f2b-wordpress (1 references)
pkts bytes target prot opt in out source destination
14M 1466M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-after-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-after-input (1 references)
pkts bytes target prot opt in out source destination
830 64740 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:137
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:138
14 632 ufw-skip-to-policy-input tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:139
590 30068 ufw-skip-to-policy-input tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:445
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ufw-skip-to-policy-input udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ufw-skip-to-policy-input all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type BROADCAST
Chain ufw-after-logging-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-after-logging-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-after-logging-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-after-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-forward (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 3
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 11
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 12
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8
0 0 ufw-user-forward all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-before-input (1 references)
pkts bytes target prot opt in out source destination
99M 534G ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
61M 20G ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
34209 1913K ufw-logging-deny all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
34209 1913K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 3
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 11
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 12
11833 788K ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68
2074K 122M ufw-not-local all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT udp -- * * 0.0.0.0/0 224.0.0.251 udp dpt:5353
0 0 ACCEPT udp -- * * 0.0.0.0/0 239.255.255.250 udp dpt:1900
2074K 122M ufw-user-input all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-before-logging-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-logging-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-logging-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-before-output (1 references)
pkts bytes target prot opt in out source destination
99M 534G ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0
62M 194G ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
499K 35M ufw-user-output all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-logging-allow (0 references)
pkts bytes target prot opt in out source destination
Chain ufw-logging-deny (2 references)
pkts bytes target prot opt in out source destination
Chain ufw-not-local (1 references)
pkts bytes target prot opt in out source destination
2074K 122M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type MULTICAST
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type BROADCAST
0 0 ufw-logging-deny all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 3/min burst 10
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-reject-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-reject-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-reject-output (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-skip-to-policy-forward (0 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-skip-to-policy-input (7 references)
pkts bytes target prot opt in out source destination
1434 95440 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-skip-to-policy-output (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-track-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-track-input (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-track-output (1 references)
pkts bytes target prot opt in out source destination
177K 13M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW
309K 20M ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW
Chain ufw-user-forward (1 references)
pkts bytes target prot opt in out source destination
Chain ufw-user-input (1 references)
pkts bytes target prot opt in out source destination
786 46748 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
53508 3142K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
475K 28M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT tcp -- * * 89.173.195.48 0.0.0.0/0 tcp dpt:3306
0 0 ACCEPT udp -- * * 89.173.195.48 0.0.0.0/0 udp dpt:3306
702 28308 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 49152:65534
0 0 ACCEPT tcp -- * * 89.173.200.197 0.0.0.0/0 tcp dpt:3306
0 0 ACCEPT udp -- * * 89.173.200.197 0.0.0.0/0 udp dpt:3306
0 0 ACCEPT tcp -- * * 46.101.206.200 0.0.0.0/0 tcp dpt:3306
0 0 ACCEPT udp -- * * 46.101.206.200 0.0.0.0/0 udp dpt:3306
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
234 288K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:443
48 2752 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:21
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:21
0 0 ACCEPT tcp -- * * 54.39.160.178 0.0.0.0/0 tcp dpt:3306
0 0 ACCEPT udp -- * * 54.39.160.178 0.0.0.0/0 udp dpt:3306
0 0 DROP all -- * * 45.33.124.193 0.0.0.0/0
0 0 DROP all -- * * 185.93.3.241 0.0.0.0/0
0 0 DROP all -- * * 209.58.131.100 0.0.0.0/0
0 0 ACCEPT tcp -- * * 188.167.252.65 0.0.0.0/0 tcp dpt:3306
0 0 ACCEPT udp -- * * 188.167.252.65 0.0.0.0/0 udp dpt:3306
0 0 DROP all -- * * 139.215.2.66 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:21
0 0 REJECT all -- * * 10.11.12.14 0.0.0.0/0 reject-with icmp-port-unreachable
Chain ufw-user-limit (0 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain ufw-user-limit-accept (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-user-logging-forward (0 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-user-logging-input (0 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-user-logging-output (0 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain ufw-user-output (1 references)
pkts bytes target prot opt in out source destination
The above list of rules is shortened, we have many other IP addresses on the list, but I ensured the structure is kept intact.
- IP 187.54.60.43 is an example of an IP ban which works properly
- IP 2.58.149.35 is somehow able to send in more requests although it uses same kind of DROP rule to get blocked
Update 2022-06-13: We found these bad requests which are let through are coming in via the CDN. Can the original IP behind the CDN be properly blocked using iptables?
Update 2022-06-20: We only allowed specific file types on our CDN. That's surely a lot simpler than any kind of firewall that can check the HTTP headers to figure out what is the original IP behind the CDN request.
