3

I have a unique Docker issue. I am developing an application which needs to connect to multiple Docker containers. The gist is, that this application will use the Docker SDK to spin up containers and connect to them as needed.

However, due to the nature of the application, we should assume that each one of these containers is compromised and unsafe. Therefore, I need to separate them from the host network (so they cannot access my devices and the WAN). I still have the constraint of needing to connect to them from my application.

It is a well-known problem that the macOS networking stack doesn't support connecting to a docker network. Normally, I'd get around this by exposing a port I need. However, this is not possible with my application, as I am using internal networks with Docker.

I'd like to accomplish something like the following. Imagine Container 2 and Container 3 are on their own private internal network. The host (which isn't a container) is controlling the Docker SDK and can query their internal IPs. Thus, it can easily connect to these machines without this network being exposed to the network of the host. Fortunately, this sort of setup works on Linux. However, I'd like to come up with a cross platform solution that works on macOS.

Network diagram

baseman101
  • 352
  • 1
  • 4
  • 19
  • Docker Desktop for Mac secretly runs a Linux VM; you might run this entire application stack inside your own VM (a) to have more direct control over the networking, and (b) to further isolate the containers from the outside environment. – David Maze Dec 06 '20 at 11:36
  • That's a good idea, and one I've thought about, but I was hoping for a better solution. Host -> VM -> Docker sounds like an awfully slow solution, but please correct me if I'm wrong. Here's something I'm putting thought into: I could containerize the application that controls the SDK, give it two networks, the host, and the internal network, and give it the socket for Docker so it can spin up other containers. Of course, it would be bad news if the container got attacked, and attackers could spin up containers using SDK, but what's the difference here than the docker host being attacked? – baseman101 Dec 06 '20 at 17:05
  • I think if your personal machine (the one whose terminal you use) is non-linux, then any docker container runs as "Host -> VM -> Docker". I have no idea how to access the VM though.. – Piyush Singh Dec 10 '20 at 14:08
  • Before going into a deep dive, did you try creating a network with the `--internal` option? Pretty sure this will prevent the host NAT rule that allows connections vai the host's 'real' interfaces https://stackoverflow.com/a/42336592/1987437 – Liam Kelly Dec 10 '20 at 14:11
  • Yes, this is what I use to isolate the container. But now, I need a way to connect to it from my macOS machine. – baseman101 Dec 13 '20 at 01:22
  • I don't see a problem using the Docker SDK directly as long as docker is running (i.e. /var/run/docker.sock) either using curl commands: curl --unix-socket /var/run/docker.sock -X GET http://localhost/v1.41/containers/json or docker-modem https://github.com/apocas/docker-modem. Basically you can hook anything to /var/run/docker.sock and Docker SDK will work. – Gillsoft AB Dec 15 '20 at 16:16
  • Remember that Mac OS X requires the userland-proxy (i.e. proxy server) so Host > userland-proxy > container server is always the case. On linux I always use "userland-proxy": false to get better performance. The daemon file should be located at ~/.docker/daemon.json. You should be able to connect to Docker subnet 192.168.65.0/24 (Netmask 255.255.255.0) or 192.168.65.0/28 (Netmask 255.255.255.240). However, if the containers are compromised then I would go for the curl variant above for minimal attack surface. – Gillsoft AB Dec 15 '20 at 16:51

1 Answers1

-1

I had a similar situation. What I ended up doing was:

  • The app manages a dynamic container-to-port mapping (just a hash table).
  • When my app (on the host) wants to launch a container, it finds an unused port in a pre-defined range (e.g. 28000-29000).
  • Once it has a port, it maps the container's port to some port in a pre-determined range (e.g. -p 28003:80).
  • When my app needs to refer to a container, it uses localhost:<port> (e.g. localhost:28001).

It turns out to not be a lot of code, but if you go that route, make sure you encapsulate the way you refer to containers (i.e. don't hard-code the hostname and port, use a class that generates the string).

All that said, you should really do some testing with a VM deployment option before you rule it out as too slow.

root
  • 5,528
  • 1
  • 7
  • 15
  • This doesn’t meet the constraint of the network being isolated from the host – baseman101 Dec 15 '20 at 15:29
  • Can you please edit your original question to clarify? IIUC they can't be completely isolated because you need the host to access the guest. Is that not the case? Or does this answer introduce a way for the guest to talk to the host that doesn't exist on that LInux setup you mention in your question? – root Dec 16 '20 at 01:35
  • My original question has: "However, due to the nature of the application, we should assume that each one of these containers is compromised and unsafe. Therefore, I need to separate them from the host network (so they cannot access my devices and the WAN)." My main concern is these devices accessing the WAN, I care less about access to the host machine, as SSH will be the only port open and public / private key authentication will be enabled. – baseman101 Dec 16 '20 at 03:10