Fail2Ban iptables entries to reject HTTPS not stopping requests to Docker container on Amazon Linux 2

Question

I've set up Fail2Ban on Amazon Linux 2, enabling the built-in nginx-http-auth jail with this override config:

[nginx-http-auth]
enabled = true
action = iptables[name=HTTPS, port=https, protocol=tcp]
logpath = <snip>/logs/*error*.log
findtime = 15m
bantime = 15m
maxretry = 5

The action is triggering and I'm getting the following entry in iptables -S:

-A f2b-HTTPS -s 120.<snip>.122/32 -j REJECT --reject-with icmp-port-unreachable

However, I can continue making new HTTPS requests from the banned IP which are receiving 401 responses from Nginx. I've replicated from two IP addresses - my phone and another EC2 host.

Here's the full output of iptables -L: (Note: Nginx is running inside Docker, as are two other containers that are isolated from the local network)

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
f2b-HTTPS  tcp  --  anywhere             anywhere             tcp dpt:https

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             ip-192-168-208-2.ap-southeast-2.compute.internal  tcp dpt:webcache

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DROP       all  -- !ip-192-168-192-0.ap-southeast-2.compute.internal/20  anywhere
DROP       all  --  anywhere            !ip-192-168-192-0.ap-southeast-2.compute.internal/20
DROP       all  -- !ip-192-168-176-0.ap-southeast-2.compute.internal/20  anywhere
DROP       all  --  anywhere            !ip-192-168-176-0.ap-southeast-2.compute.internal/20
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain f2b-HTTPS (1 references)
target     prot opt source               destination
REJECT     all  --  120.<snip>.122       anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere

Why isn't the iptable rule stopping HTTPS requests?

Do I need to change my fail2ban config somehow to make it work?

Answer 1

As noted by @tater in the comments above, fail2ban inserts itself into the INPUT chain by default, but traffic to Docker containers is routed using the FORWARD chain, which is routed without touching the INPUT chain. You can see that here:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
f2b-HTTPS  tcp  --  anywhere             anywhere             tcp dpt:https
...

We can test getting fail2ban to work with Docker by inserting a similar fail2ban rule into the FORWARD chain, like this (NOTE: This is not the long term solution):

$ sudo iptables -I FORWARD 1 -p tcp -j f2b-HTTPS

In english, this command says "Insert into the FORWARD chain, at position 1 (i.e. the first rule), for all traffic on the TCP protocol, a reference to the f2b-HTTPS chain", which has the effect of including all that chain's rules at that position.

After doing that, the FORWARD chain should contain the new rule at the top:

$ sudo iptables -L
...
Chain FORWARD (policy DROP)
target     prot opt source               destination
f2b-HTTPS  tcp  --  anywhere             anywhere
DOCKER-USER  all  --  anywhere             anywhere
...

The rules that fail2ban automatically manages under the f2b-HTTPS chain are then used to reject traffic destined for Docker.

However, we want fail2ban to do this for us automatically, rather than having to create our own iptables rules. The solution, then, is to add a second action to the jail config in the file under jail.d/:

action = iptables[actname=iptables-input,   name=HTTPS,                       port=https, protocol=tcp]
         iptables[actname=iptables-forward, name=HTTPS-DOCKER, chain=FORWARD, port=8080, protocol=tcp]

It would probably suffice to just add chain=FORWARD to my original rule, but I decided to keep the INPUT rule as well.

NOTE: The port in the iptables-forward rule is 8080 because that is where my Docker container is listening, and it's the destination forward port that's matched by iptables on a FORWARD rule (it seems), not the inbound port.

Two other things I discovered while solving this:

1. fail2ban isn't enabled by default when it's installed. To enable it, run:

sudo systemctl enable fail2ban

2. If your system restarts while a ban is active, fail2ban will insert its rules at the top of the FORWARD chain before Docker inserts its rules at the top of the FORWARD chain. This means that the Docker rules will override the fail2ban rules and bans won't work. (You can simulate this by just doing sudo service docker restart.) To overcome this, you need to make fail2ban start after Docker has established its networking. I did this by changing '/etc/lib/systemd/system/fail2ban.service' by:

• Adding 'docker.service' to the list of 'After:'
• Removing the PartOf=firewalld.service line
• Adding this in the [Service] section to wait for port 443 to open: ExecStartPre=/bin/bash -c '(while ! nc -z -v -w1 localhost 443 > /dev/null; do echo "Waiting for port 443 to open..."; sleep 2; done); sleep 2'
• (NOTE: You also need nc installed)