Counting bandwidth from a Docker container

Question

I'm trying to figure out how to track bandwidth coming from a Docker container.

Normally I use --uid-owner as mark to keep track of bandwidth usage for a given user. However, even when I run all the processes as the user inside the docker container --uid-owner is not working. Instead of using --uid-owner, I tried just tracking all packets coming from the virtual ethernet device that docker creates.

This, however, ended up doing nothing too: no matter what I try, no packets are being caught.

Out of sheer desperation, I tried just putting the rules in all the chains but had no result neither.

Chain PREROUTING (policy ACCEPT 3041 packets, 7849454 bytes)
num      pkts      bytes target     prot opt in     out     source               destination
1           0        0 MARK       tcp  --  veth5a36 any     anywhere             anywhere             MARK set 0x1

Chain INPUT (policy ACCEPT 273 packets, 23305 bytes)
num      pkts      bytes target     prot opt in     out     source               destination
1           0        0 MARK       tcp  --  veth5a36 any     anywhere             anywhere             MARK set 0x1

Chain FORWARD (policy ACCEPT 2750 packets, 7821109 bytes)
num      pkts      bytes target     prot opt in     out     source               destination
1           0        0 MARK       tcp  --  any    veth5a36  anywhere             anywhere             MARK set 0x1
2           0        0 MARK       tcp  --  veth5a36 any     anywhere             anywhere             MARK set 0x1
3           0        0            all  --  veth5a36 eth0    anywhere             anywhere             mark match 0x1

Chain OUTPUT (policy ACCEPT 293 packets, 80020 bytes)
num      pkts      bytes target     prot opt in     out     source               destination
1           0        0 MARK       tcp  --  any    veth5a36  anywhere             anywhere             MARK set 0x1

Chain POSTROUTING (policy ACCEPT 3043 packets, 7901129 bytes)
num      pkts      bytes target     prot opt in     out     source               destination
1           0        0 MARK       tcp  --  any    veth5a36  anywhere             anywhere             MARK set 0x1

Can anybody tell me how to successfully mark packages from a docker container? Preferably using --uid-owner but I will take anything at this point :)

Answer 1

The problem is related to namespaces. Docker uses them to isolate resources, which also means they don't count to the host's totals.

When you run iptables on the host, you're basically only looking into the host's namespace, and the packets you are interested in are tallied to the container's namespace. To work around this issue you can use ip netns to still run iptables on the host, but in the container's network namespace.

First, ip netns has a somewhat counter intuitive interface. In order to attach to the namespace of an existing process (in this case your container), you have to create a link in /var/run/netns/ to the process' namespace:

# ln -sf /proc/`docker inspect -f '{{ .State.Pid }}' YOUR_CONTAINER`/ns/net /var/run/netns/SOME_NAME

(you may have to mkdir /var/run/netns)

Now you're free to run iptables in your container's namespace:

# ip netns exec SOME_NAME iptables -L -nv

Note that this outputs iptables' ruleset inside the container, which will probably be empty.

If you're currently using --uid-owner just to have per-user counters, you don't even need it anymore, because in this case the chain counters apply only to the container and should then be enough.

Finally, you can cleanup /var/run/netns.

Multiple containers per user

If you have multiple containers per user and want to tally them together, you can start your containers with --net=container:OTHER_CONTAINER_FROM_USER, so that their namespaces are merged.

This has the disadvantage of merging all aspects of the network stack, including open ports, so you can't have two containers for the same user listening on the same port.

If this is a prohibitive limitation, you can tally the containers individually and group based on uid later.

You can find some more information about this issue here.

Answer 2

Not exactly what you asked, but I figure it will get the job done.

Quote from the Docker blog:

Interface-level counters

Since each container has a virtual Ethernet interface, you might want to check directly the TX and RX counters of this interface.

[...]

But for now, the best way is to check the metrics from within the containers. I’m not talking about running a special agent in the container, or anything like that. We are going to run an executable from the host environment, but within the network namespace of a container.

The exact format of the command is:

ip netns exec <nsname> <command...>

For instance:

ip netns exec mycontainer netstat -i