iptables is marking packets with wrong number while copying file by scp

Question

[Case1] When copying file from server (being on client) to client, packets are marked incorrect with 3 (root).

[Case2] When copying file from server (being on server) to client, packets are marked correct with 1003 (test1).

Server ip 192.168.0.16, test1 is server user.

Client ip 192.168.0.10, client is client user.

[Case1]

client@192.168.0.10:~$ scp -P 22 test1@192.168.0.16:/home/test1/archlinux-bootstrap-2016.03.01-x86_64.tar.gz /tmp/

ps check, while copying:

test1@192.168.0.16:~$ ps aux | grep scp
root      1653  0.1  0.0  32668  4408 ?        Ss   19:31   0:00  \_ sshd: test1 [priv]
test1      1655  3.0  0.0  36104  6912 ?        S    19:31   0:00      \_ sshd: test1@notty
test1      1656  1.3  0.0  27516  2648 ?        Ss   19:31   0:00          \_ scp -f /home/test1/archlinux-bootstrap-2016.03.01-x86_64.tar.gz

bmon [class 1:3 is a root user] [class 1:1003 is a test1 user]

  imq0 (outgoing)              │   1.17MiB    818      │   1.12MiB    781
    qdisc 1: (htb)             │      0         0      │   1.12MiB    781
      cls :3 (fw)              │      0         0      │      0         0
      cls :3eb (fw)            │      0         0      │      0         0
      class 1:1 (htb)          │      0         0      │   1.12MiB    781   99%
        class 1:2 (htb)        │      0         0      │    430B        1    0%
        class 1:3 (htb)        │      0         0      │   1.12MiB    780   208%
        class 1:1003 (htb)     │      0         0      │      0         0    0%
        class 1:5 (htb)        │      0         0      │      0         0    0%
        class 1:6 (htb)        │      0         0      │      0         0    0%

[Case2]

test1@192.168.0.16:~$ scp -P 22 archlinux-bootstrap-2016.03.01-x86_64.tar.gz client@192.168.0.10:~/

ps check, while copying:

test1@192.168.0.16:~$ ps aux | grep scp
root      1637  0.0  0.0  32668  4400 ?        Ss   19:29   0:00  \_ sshd: test1 [priv]
test1      1639  0.0  0.0  32668  3240 ?        S    19:30   0:00      \_ sshd: test1@pts/3
test1      1640  0.0  0.0  20540  3296 pts/3    Ss   19:30   0:00          \_ -bash
test1      1650  0.0  0.0  27516  2640 pts/3    S+   19:30   0:00              \_ scp -P 22 archlinux-bootstrap-2016.03.01-x86_64.tar.gz client@192.168.0.10:~/
test1      1651  0.0  0.0  30636  6748 pts/3    S+   19:30   0:00                  \_ /usr/bin/ssh -x -oForwardAgent=no -oPermitLocalCommand=no -oClearAllForwardings=yes -p 22 -l

bmon [class 1:3 is a root user] [class 1:1003 is a test1 user]

  imq0                         │ 142.83KiB    103      │  98.50KiB     68
    qdisc 1: (htb)             │      0         0      │  98.50KiB     68
      cls :3eb (fw)            │      0         0      │      0         0
      cls :3 (fw)              │      0         0      │      0         0
      class 1:1 (htb)          │      0         0      │  98.50KiB     68    1%
        class 1:2 (htb)        │      0         0      │    533B        2    0%
        class 1:3 (htb)        │      0         0      │      0         0    0%
        class 1:1003 (htb)     │      0         0      │  97.97KiB     66   100%
        class 1:5 (htb)        │      0         0      │      0         0    0%

Iptables rules:

# IN
  iptables -t mangle -A PREROUTING -i eth0 -j IMQ --todev 1
  iptables -t mangle -A PREROUTING -j CONNMARK  --restore-mark
# OUT
  iptables -t mangle -N IMQ-OUT
  iptables -t mangle -A POSTROUTING -o eth0 -j IMQ-OUT
  iptables -t mangle -A IMQ-OUT -o eth0 -m owner --uid-owner root -j MARK --set-mark 3
  iptables -t mangle -A IMQ-OUT -o eth0 -m owner --uid-owner root -j RETURN
  iptables -t mangle -A IMQ-OUT -o eth0 -m owner --uid-owner test1 -j MARK --set-mark 1003
  iptables -t mangle -A IMQ-OUT -o eth0 -m owner --uid-owner test1 -j RETURN
  iptables -t mangle -A POSTROUTING -j CONNMARK  --save-mark
  iptables -t mangle -A POSTROUTING -o eth0 -j IMQ --todev 0

Could someone explain me why in [Case1] server thinks that outgoing connection is root connection even if ps shows user test1?

Right now I'm trying to use cgroup to shape traffic, by user.

Same situation with cgroup.

echo '1003' > /cgroup/cpu_mem_blkio/users/test1/net_cls.classid
iptables -t mangle -A IMQ-OUT -o eth0 -m cgroup --cgroup 1003 -j MARK --set-mark 1003

Answer 1

In [Case1], the socket that connects sshd to the client during the file transfer doesn't belong to the scp process. Instead, it belongs to a sshd process that is owned by root. So, the rule that sets the mark 3 prevails.

---

See my example. First, I launch a copy task from the client:

[amg1127@amg1127-laptop /tmp]$ scp -4 amg1127-sala:/tmp/virtualbox-machine.vdi .
virtualbox-machine.vdi                        0% 5488KB   1.4MB/s   55:47 ETA^

Then, I figure in the server what processes and sockets are related to the copy.

[root@amg1127-sala /tmp]# lsof /tmp/virtualbox-machine.vdi
COMMAND   PID    USER   FD   TYPE DEVICE   SIZE/OFF     NODE NAME
scp     12107    1000    3r   REG  254,5 4824498176 20439792 /tmp/virtualbox-machine.vdi

[root@amg1127-sala /tmp]# netstat -pan | grep ':22'
tcp   0      0 0.0.0.0:22         0.0.0.0:*           LISTEN      1029/sshd           
tcp   0 195936 192.168.254.10:22  192.168.254.9:59574 ESTABLISHED 12089/sshd: amg1127 
tcp6  0      0 :::22              :::*                LISTEN      1029/sshd           

[root@amg1127-sala /tmp]# ps -C sshd -C scp -o uid,euid,pid,ppid,cmd
 UID  EUID   PID  PPID CMD
   0     0  1029     1 /usr/bin/sshd -D
   0     0 12089  1029 sshd: amg1127 [priv]
1000  1000 12106 12089 sshd: amg1127@notty
1000  1000 12107 12106 scp -f /tmp/virtualbox-machine.vdi

Note that:

1. lsof shows the process that reads the /tmp/virtualbox-machine.vdi file is a scp process whose PID is 12107 and owner is 1000.
2. netstat shows the socket that sends the file data to the remote client is a sshd process whose PID is 12089 and owner is 0.

Both processes are probably exchanging data through anonymous pipes.