I want to run multiple containers, each of which sftp's into a server, and downloads file. I want to manipulate qdiscs to give priority to a certain container.
The containers are attached to the docker0 bridge by default and therefore are an assigned an address on the 172.17.0.X subnet such as 172.17.0.4.
I created the following tc commands for the docker0 interface.
#Delete an previously intalled qdisck on docker0
tc qdisc del dev docker0 root
# Install a classless TBF qdisc, which becomes a parent to PRIO qdisc
# Having a TBF as a parent qdisc helps in bandwith sharing
tc qdisc add dev docker0 root handle 1: tbf rate 100mbps burst 1600 limit 1
# Install classful PRIO disc with 3 classes/bands
tc qdisc add dev docker0 parent 1:1 handle 2: prio bands 3
# Install SFQ qdisc on each of the classes of PRIO disc
tc qdisc add dev docker0 parent 2:1 handle 10 sfq perturb 20
tc qdisc add dev docker0 parent 2:2 handle 20 sfq perturb 20
tc qdisc add dev docker0 parent 2:3 handle 30 sfq perturb 20
# Add a filter rule to TC prio qdisc based on the IP address of a container.
tc filter add dev docker0 protocol ip parent 2:0 prio 1 u32 match ip dst 172.17.0.4/32 flowid 2:1
tc filter add dev docker0 protocol ip parent 2:0 prio 1 u32 match ip dst 172.17.0.5/32 flowid 2:2
tc filter add dev docker0 protocol ip parent 2:0 prio 1 u32 match ip dst 172.17.0.6/32 flowid 2:3
Here, I create three levels of flow priority based on the destination. In the end, I want to track the throughput that the containers are able to achieve over time after the rules are implemented. I am using the nethogs tool to track the amount of bandwith/throughput on each docker container interface at each interval (each second for example). For example, sudo nethogs -t &> nethogs.log & command can trace in the background and output bandwith results nethogs file.
However, based my understanding of the tc rules, traffic should be prioritized to the container with the static ip address of 172.17.0.4 first. This means that the throughput for that container should be the highest until the download is finished.
I am running the container with
docker run --cap-add=NET_ADMIN -it sftp-image
which containers scripts to sftp into a server and download a file. I made sure that all containers executed the sftp command at the same with the ENTRYPOINT value of
sleep $((60 - $(date +%s) % 60)) && date && ./root/sftp.sh
After running three docker containers, with address of 172.17.0.4, 172.17.0.5, and 172.17.0.6, and each downloading a file of 500mb. The bandwidth at each interval seems is the same, and the time they finish downloading is almost the same.
After a digging deeper, the tc filter show dev docker0 didn't output any results. This means that the filters weren't actually added. To confirm this,
$ tc -s -d qdisc show dev docker0
qdisc tbf 1: root refcnt 2 rate 800Mbit burst 1600b/1 mpu 0b limit 1b linklayer ethernet
Sent 495078187 bytes 328558 pkt (dropped 0, overlimits 183436 requeues 0)
backlog 3882034592b 0p requeues 0
qdisc prio 2: parent 1:1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 495078187 bytes 328558 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc sfq 30: parent 2:3 limit 127p quantum 1514b depth 127 flows 128 divisor 1024 perturb 20sec
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc sfq 10: parent 2:1 limit 127p quantum 1514b depth 127 flows 128 divisor 1024 perturb 20sec
Sent 495077573 bytes 328546 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc sfq 20: parent 2:2 limit 127p quantum 1514b depth 127 flows 128 divisor 1024 perturb 20sec
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
Which means that packets were not actually filtered to the correct PRIO band. And all packets simply went to the first transmission queue with SFQ qdisc of handle 10.
QUESTION How would I properly add filter's on the docker0 to achieve what I want?
PS.
Here is the ./sftp file
#!/bin/bash
USER=root
HOST=<SERVER_IP>
PASS=<SERVER_PASS>
PORT=<SERVER_PORT>
REMOTE_FILE=output500Mb.dat
LOCAL_FILE=$REMOTE_FILE
ip addr show eth0
date
echo Start downloading
sshpass -p $PASS sftp -o StrictHostKeyChecking=no -P $PORT $USER@$HOST << EOF
cd /filepath/
get $REMOTE_FILE
EOF
date
echo Done downloading
Other important information:
$ uname -r
5.15.0-71-generic
$ tc -V
tc utility, iproute2-ss200127
EDIT 1 I attempted the method shown here https://bugzilla.redhat.com/show_bug.cgi?id=1297230, by using an ifb0 interface to redirect docker0 traffic to ifb0 and then try to shape the egress traffic there.
tc qdisc add dev docker0 handle ffff: ingress
tc filter add dev docker0 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
$ sudo tc filter show dev docker0 root
filter parent ffff: protocol ip pref 49152 u32 chain 0
filter parent ffff: protocol ip pref 49152 u32 chain 0 fh 800: ht divisor 1
filter parent ffff: protocol ip pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw
match 00000000/00000000 at 0
action order 1: mirred (Egress Redirect to device ifb0) stolen
index 1 ref 1 bind 1
So adding filters seems possible. I tried adding the same tc filters to ifb0 (instead of docker0) but still could not get
tc filter add dev ifb0 protocol ip parent 2:0 prio 1 u32 match ip dst 172.17.0.4/32 flowid 2:1
tc filter add dev ifb0 protocol ip parent 2:0 prio 1 u32 match ip dst 172.17.0.5/32 flowid 2:2
tc filter add dev ifb0 protocol ip parent 2:0 prio 1 u32 match ip dst 172.17.0.6/32 flowid 2:3
With this method I could also no longer get ping responses, this redirection messed with outgoing traffic. But at the same time, the filters still could not be added, which doesn't solve my key problem.
EDIT 2 If I don't add the classless TBF qdisc, it seems I can add the filters
$ tc filter show dev docker0
filter parent 1: protocol ip pref 1 u32 chain 0
filter parent 1: protocol ip pref 1 u32 chain 0 fh 800: ht divisor 1
filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 not_in_hw
match ac110000/ffff0000 at 16
filter parent 1: protocol ip pref 2 u32 chain 0
filter parent 1: protocol ip pref 2 u32 chain 0 fh 801: ht divisor 1
filter parent 1: protocol ip pref 2 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:2 not_in_hw
match ac110000/ffff0000 at 16
filter parent 1: protocol ip pref 3 u32 chain 0
filter parent 1: protocol ip pref 3 u32 chain 0 fh 802: ht divisor 1
filter parent 1: protocol ip pref 3 u32 chain 0 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:3 not_in_hw
match ac110000/ffff0000 at 16
However, priority doesn't seem to be given to the first container with the 172.17.0.4 address. I can see the packets being routed to the proper prio. In theory, if given priority, that means the first container should have the highest throughput in the beginning until it finishes downloading, then the second container, then the third.
