I'm porting an application to k8s. The application currently consists of pairs of dockers, Trusted and Untrusted, where Trusted is connected to the regular bridge network and talks to internal services, while Untrusted is connected to a separate network allowing access only to external IPs. Untrusted accesses the network according to user-generated data, hence must have internet access and must not be able to access internal IPs.
Trusted and Untrusted communicate using a pair of FIFOs since they run on the same machine (Using a Unix Domain Socket was ~20% slower, I didn't yet test local TCP/IP but I suspect a bigger performance hit - the service is not entirely horizontally scalable due to ordering constraints so single-machine performance matters).
I've hit a wall porting this setup to k8s: the original idea was to use a Pod for each pair of dockers and using emptyDir to share the FIFOs, but there seems to be no way to create separate network limitations for a single container in a Pod since they all share a networking container. Is there a way to do this?
What's the alternative if this isn't possible? Setting up the untrusted containers in a separate namespace and applying a limited network access policy allowing only tightly-controlled access to the rest of the cluster?
