nsjail process does not have network access

Question

We run our java software builds on Linux docker containers and I modified our integration testing pipeline to wrap the test execution inside nsjail, this looks something like this:

nsjail -Mo --chroot / --user root --group root --disable_rlimit -- /bin/sh -c "our integration test execution goes here"

The reason is to isolate the network for these tests so that they can run in parallel and still bind to potentially the same ports. This works really well and the performance increase is great.

But (of course there is a but) I don't have any outgoing network access in the jailed process. I was under the impression that the jailed process would have some sort of bridged (not sure that's the term) networking, but I appear to be mistaken.

Is there any way to have this sort of networked access for an nsjailed process?

score 1 · Answer 1 · answered May 08 '21 at 01:34

1

You might take a look at https://github.com/google/nsjail/blob/master/configs/firefox-with-net.cfg which contains a configuration that allows firefox to run inside a jail but with internet access.

answered May 08 '21 at 01:34

andref

111
2

lucidbrot · Answer 2 · 2022-11-02T09:13:18.100

This is only a partial answer as I do not really understand the differences between the options. It outlines options but for me, only the last one worked.

nsjail --help shows us

  --macvlan_iface|-I VALUE
    Interface which will be cloned (MACVLAN) and put inside the subprocess' namespace as 'vs'
  --macvlan_vs_ip VALUE
    IP of the 'vs' interface (e.g. "192.168.0.1")
  --macvlan_vs_nm VALUE
    Netmask of the 'vs' interface (e.g. "255.255.255.0")
  --macvlan_vs_gw VALUE
    Default GW for the 'vs' interface (e.g. "192.168.0.1")
    # I guess that "GW" means Gateway
  --macvlan_vs_ma VALUE
    MAC-address of the 'vs' interface (e.g. "ba:ad:ba:be:45:00")

You can find an example on the nsjail github. Visit the link for the output. Notice that this "requires root/setuid".

 sudo ./nsjail --user 9999 --group 9999 --macvlan_iface eth0 --chroot /chroot/ -Mo --macvlan_vs_ip 192.168.0.44 --macvlan_vs_nm 255.255.255.0 --macvlan_vs_gw 192.168.0.1 -- /bin/sh -i

These parameters have some defaults:

    /* Parameters for the cloned MACVLAN interface inside jail */
    optional string macvlan_iface = 83; /* Interface to be cloned, eg 'eth0' */
    optional string macvlan_vs_ip = 84 [default = "192.168.0.2"];
    optional string macvlan_vs_nm = 85 [default = "255.255.255.0"];
    optional string macvlan_vs_gw = 86 [default = "192.168.0.1"];
    optional string macvlan_vs_ma = 87 [default = ""];
    optional string macvlan_vs_mo = 88 [default = "private"];

so we mainly need to care about setting --macvlan_iface eth0 with the correct interface in place of eth0, and modifying any of the defaults if we care about them.

However, this does not always work. Particularly there is a currently open issue where it is stated that

From my very coarse understanding of macvlan, it doesn't seem possible to use it if the adapter is virtualized or otherwise allow for multiple MAC addresses (e.g. on VMWare or a cloud hosting provider).

This might explain why I could not get it to work inside my docker container even with --privileged --add-cap="NET_ADMIN" docker flags and keep:caps: true in the nsjail config.

There is also the option --iface_own which fails in similar ways inside my docker container.

The --help output also shows the option

  --disable_clone_newnet|-N 
    Don't use CLONE_NEWNET. Enable global networking inside the jail

which is also used in an nsjail example with internet (as already pointed out by andref's link-only answer). It turns out that in the config file the syntax is just clone_newnet: false.

Launching the nsjail works this time, but there still does not seem to be name resolution inside the jail. However, if we also add the /etc/resolv.conf mount like in the linked example, it works!

clone_newnet: false

mount  {
    src_content: "nameserver 8.8.8.8"
    dst: "/etc/resolv.conf"
  }

However, note that in this case there will not be any separation between the ports you're listening on from different nsjail instances.

I managed to make outgoing network access work for a dynamic number of jails and without requiring one LAN ip address per jail. I did so using veth pairs and masquerading. I hope I will find time to write up how to do so once I have thoroughly tested it — lucidbrot, Nov 19 '22 at 18:33

nsjail process does not have network access

2 Answers2