3

Istio on Kubernetes injects an Envoy sidecar to run alongside Pods and implement a service mesh, however Istio itself cannot ensure traffic does not bypass this proxy; if that happens Istio security policy is no longer applied.

Therefore, I am trying to understand all ways in which this bypass could happen (assuming Envoy itself hasn't been compromised) and find ways to prevent them so that TCP traffic originating from a Pod's network namespace is guaranteed to have gone through Envoy (or at least is much more likely to have done):

  1. Since (at the time of writing) Envoy does not support UDP (it's nearly there), UDP traffic won't be proxied, so use NetworkPolicy to ensure only TCP traffic is allowed to/from the Pod (e.g. to avoid TCP traffic being tunnelled out via a VPN over UDP)
  2. Drop NET_ADMIN capability to prevent the Pod from reconfiguring the IPTables rules in its network namespace that capture traffic
  3. Drop NET_RAW capability to prevent the Pod from opening raw sockets and bypassing the netfilter hook points that IPTables makes use of

The only other attack vector I know of would be a kernel vulnerability - are there any others? Maybe there are other L3/4 protocols that IPTables doesn't recognise or ignores?

I understand that eBPF and Cilium could be used to enforce this interception at the socket level, but I am interested in the case of using vanilla Istio on Kubernetes.

EDIT: I am also assuming the workload does not have Kubernetes API server access

dippynark
  • 2,743
  • 20
  • 58

2 Answers2

1

Envoy is not designed to be used as a firewall. Service meshes that rely on it such as Istio or Cilium only consider it a bug if you can bypass the policies on the receiving end.

For example, any pod can trivially bypass any Istio or Cilium policies by terminating its own Envoy with curl localhost:15000/quitquitquit and starting a custom proxy on port 15001 that allows everything before Envoy is restarted.

You can patch up that particular hole, but since resisting such attacks is not a design goal for the service meshes, there are probably dozens of other ways to accomplish the same thing. New ways bypass these policies may also be added in subsequent releases.

If you want your security policies to be actually enforced on the end that initiates the connection and not only on the receiving end, consider using a network policy implementation for which it is a design goal, such as Calico.

Shnatsel
  • 4,008
  • 1
  • 24
  • 25
  • AFAIK you can use Cilium to implement Kubernetes Network policies which can white-list traffic allowed to specific pods, to be restricted to specific ports, independent of what the service mesh is doing. (https://cilium.io/blog/2018/09/19/kubernetes-network-policies/) – Rory McCune Nov 21 '19 at 13:01
  • It's possible that the example you linked works because the policies are enforced on the receiving end of the connection. I'm not 100% sure about Cilium, but with Istio you *definitely* cannot rely on your pod's Envoy actually enforcing anything. (Source: I've talked to an Istio developer about this). – Shnatsel Nov 21 '19 at 13:08
  • This is great `curl -X POST 127.0.0.1:15000/quitquitquit` disables the proxy for me. I feel this point/possibility should be made clearer in the docs (at least I haven't come across this before). – dippynark Nov 21 '19 at 16:02
-1

Envoy is relatively simple to bypass and Cilium is using envoy just like Istio. So it will not be able to prevent bypassing envoy's upstream.

Both Istio and Cilium have sites listing CVE's about security vulnerabilities.

From within the control plane it is possible to affect sidecar injection or iptables rules with annotations so once someone gets access to cluster admin privileges there is no defense.

You can use calico to lock down communication so the only traffic that flows is the traffic you want to flow.

Calico also offers seamless integration with Istio to enforce network policy within the Istio service mesh.

Of course applications and services in pods should also be designed with best practices for security measures in mind.

Update:

To clarify I suggested calico for a zero trust network model. Without it You can mess with envoy from the app pod since they are on the same network as the admin interface. So locking down communication between app pod and admin interface is vital vulnerability fix.

Even without privileges to cluster-admin you can affect envoy from app pod with just curl command.

Piotr Malec
  • 3,429
  • 11
  • 16
  • 1
    I don't think this is all correct or relevant 1) Cilium's CNI plugin *does* help prevent bypass using eBPF and sockmaps by intercepting TCP socket creation (see the linked video) 2) the CVEs you reference have been fixed 3) you can't elevate privileges to cluster-admin if you don't have them already without some serious k8s vulnerability (and anyway you have worse problems in that case) - I am talking about a normal workload that does not have API server access (should have clarified, but thought it was clear from the listed attacks) 4) Calico is not the only network policy implementation – dippynark Nov 20 '19 at 18:59