iptables with docker port mapping

Question

Iptables rules are notoriously difficult to set up when Docker is running on the host, and I thought I had a definitive solution in this fantastic blog post: https://unrouted.io/2017/08/15/docker-firewall/

The configuration described in this blog post has served me well for a long time, but I'm now facing a problem I never had before.

I am running a docker container that exposes a service on port 465 on the host. Port 465 maps to port 25 in the container. Here's how to simulate such a service:

$ docker run --rm -it -p 465:25 python:3.6 python3 -m http.server 25

My problem is that I cannot access port 465 on my server from the outside:

$ curl mydomain.com:465
curl: (7) Failed to connect to mydomain.com port 465: No route to host

However, and here comes the interesting part, I do manage to access the service if the port on the host maps to the same port in the container. In other words, when I run on the host:

$ docker run --rm -it -p 465:465 python:3.6 python3 -m http.server 465

then I can access the service from the outside:

$ curl mydomain.com:465
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org...

This whole problem is due to my iptables definition: I know that because when I flush the iptables rules, I do manage to access the service from outside, whatever the port mapping.

Here are my iptable rules:

*filter
# Source: https://unrouted.io/2017/08/15/docker-firewall/
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]

-F INPUT
-F DOCKER-USER
-F FILTERS

-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -s 127.0.0.0/8 -j REJECT
-A INPUT -p icmp --icmp-type any -j ACCEPT
-A INPUT -j FILTERS
-A DOCKER-USER -i eth0 -j FILTERS
-A FILTERS -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A FILTERS -m state --state NEW -m tcp -p tcp --dport 465 -j ACCEPT

-A FILTERS -j REJECT --reject-with icmp-host-prohibited
COMMIT

How should I modify my iptables so that I can access my container from the outside, whatever the port mapping?

EDIT:

Here are the complete iptables rules in the failing scenario (465:25 mapping):

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  loopback/8           anywhere             reject-with icmp-port-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp any
FILTERS    all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination        
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        

Chain DOCKER (3 references)
target     prot opt source               destination        
ACCEPT     tcp  --  anywhere             172.19.0.4           tcp dpt:3000
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:smtp

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination        
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
target     prot opt source               destination        
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination        
FILTERS    all  --  anywhere             anywhere            

Chain FILTERS (2 references)
target     prot opt source               destination        
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:urd
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

score 5 · Accepted Answer · answered Feb 20 '19 at 21:03

Thanks for reaching out to me on Twitter. I've actually looked into this issue before without someone else who noticed it and I think I know whats happening. In your example:

docker run --rm -it -p 465:25 python:3.6 python3 -m http.server 25

If you look at your full firewall config with iptables-save you'll see a bunch of NAT rules. You'll probably see something that looks like this in the *nat section:

-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
... snip ...
-A DOCKER ! -i br-abbaabbaabba -p tcp -m tcp --dport 465 -j DNAT --to-destination 172.18.0.10:25

So this rule is executed in the PREROUTING phase and rewrites the incoming packet to look like it was always for port 25 and not port 465. And this happens before the filter tables INPUT chain runs.

So I reckon if you allowed traffic to port 25, then actually you would be able to access port 465 too. Obviously you don't want to allow access to all port 25 because that includes your host's port 25.

All the usual tricks you'd do at this point are made that much more difficult because of Docker.

Option 1

You could go down the explicit is better than implicit route and split the host vs docker rules:

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:FILTERS - [0:0]
:DOCKER-USER - [0:0]

-F INPUT
-F DOCKER-USER
-F FILTERS

-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -s 127.0.0.0/8 -j REJECT
-A INPUT -p icmp --icmp-type any -j ACCEPT

# Rules for services running on the host:
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited

# Rules for services running in containers:
-A DOCKER-USER -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
# This says dport 25, but is actually 465. Yay for prerouting + NAT.
# Service on real host port 25 should still be inaccessible because DOCKER-USER
# is only accessible via `FORWARD` and not `INPUT`...
-A DOCKER-USER -i eth0 -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A DOCKER-USER -j REJECT --reject-with icmp-host-prohibited

COMMIT

It's still unsatisfying that you are allowing traffic to port 25..

Option 2

I believe right now Docker doesn't put anything in *raw or *mangle so its safe to add your own rules there. Obviously there are limitations with these tables (raw is before connection tracking, mangle is only for marking connections) so thats not great either.

Option 3

Finally, the only other thing I can think the conntrack iptables module might have the answer with --ctorigdstport, but i've never tried it myself. Looking at this you could try:

iptables -A FILTERS -p tcp --dport 25 -m conntrack --ctstate NEW --ctorigdstport 465 -j ACCEPT

A bit ugly to look at, but explicit about what is happening. If you try this one and it works let me know and i'll see about writing it up / updating that blog post.

I can confirm that both options 1 and 3 work \o/ I haven't tried option 2 because I frankly did not understand it. With option 3 it's still a bit disappointing that iptables needs to understand the inside port of my container to function but I'll be satisfied anyway. Thanks for taking the time to answer! — Régis B., Feb 26 '19 at 18:09
Indeed it is. It's tempting to always use `--ctorigdstport` and not bother providing `--dport`. But be mindful of security (explicit is probably more secure if you don't thoroughly understand iptables) and performance (there is a small overhead for using conntrack) — Jc2k, Feb 27 '19 at 19:19

iptables with docker port mapping

1 Answers1

Option 1

Option 2

Option 3