5

I am looking at how to make OpenVPN client work on a pod's container, I explain what I do, but you can skip all my explanation and offer your solution directly, I don't care replacing all the below with your steps if it works, I want to make my container to use a VPN (ExpressVPN for example) in a way that both external and internal networking works.

I have a docker image that is an OpenVPN Client, it works fine with the command:

docker run --rm -it --cap-add=NET_ADMIN --device=/dev/net/tun my-app /bin/bash

The docker image had an entry point bash script:

curl https://vpnvendor/configurations.zip -o /app/configurations.zip
mkdir -p /app/open_vpn/ip_vanish/config
unzip /app/configurations.zip -d /app/open_vpn/config
printf "username\npassword\n" > /app/open_vpn/vpn-auth.conf
cd /app/open_vpn/config
openvpn --config ./config.ovpn --auth-user-pass /app/open_vpn/vpn-auth.conf

It works fine, but when I deploy it as a container in a K8S Pod, it breaks, it is understandable, K8S clusters need internal network communication between the nodes, so the VPN breaks it ... how do I make it work? the Google search was frustrating, none of the solutions worked and there were just a few, there is one with similar issue: OpenVPN-Client Pod on K8s - Local network unreachable But did not understand it very well, please help.

Since IPVanish is well known, let's take their ovpn example, I use other vendor but had access to an IPVanish account and it does not work either:

client
dev tun
proto udp
remote lon-a52.ipvanish.com 443
resolv-retry infinite
nobind
persist-key
persist-tun
persist-remote-ip
ca ca.ipvanish.com.crt
verify-x509-name lon-a52.ipvanish.com name
auth-user-pass
comp-lzo
verb 3
auth SHA256
cipher AES-256-CBC
keysize 256
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-DSS-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA

I accept responses in Golang or YAML it does not matter, although I use go-client, my code for pod creation is:

podObj := &v1.Pod{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "mypod",
            Namespace: "default",
        },
        Spec: v1.PodSpec{
            Containers: []v1.Container{
                {
                    Name:            "worker1",
                    Image:           "192.168.1.138:5000/myimage",
                    ImagePullPolicy: v1.PullAlways,
                    Stdin: true,
                    TTY:   true,
                    /* Trying to simulate --device=/dev/net/tun I copied the below, but it does not work
// https://garunski.medium.com/openvpn-and-minikube-25511099f8de
                    VolumeMounts: []v1.VolumeMount{
                        {
                            ReadOnly:  true,
                            Name:      "dev-tun",
                            MountPath: "/dev/net/tun",
                        },
                    },*/
                    SecurityContext: &v1.SecurityContext{
                        // Taken from https://caveofcode.com/how-to-setup-a-vpn-connection-from-inside-a-pod-in-kubernetes/
                        Privileged: boolPtr(true),
                        Capabilities: &v1.Capabilities{
                            Add: []v1.Capability{
                                "NET_ADMIN",
                            },
                        },
                    },
                },
            },
            NodeName: "worker-node01",
        },
    }
clientset.CoreV1().Pods("default").Create(context.Background(), podObj, metav1.CreateOptions{})

I can add the NET_ADMIN capability, but I need also to give access to the /dev/net/tun device and that's the problem, but even If I find a way, it will break internal networking.

Update one

I made external networking work, by adding the following two lines in my docker's entry point:

# Taken from https://caveofcode.com/how-to-setup-a-vpn-connection-from-inside-a-pod-in-kubernetes/
mknod /dev/net/tun c 10 200
chmod 600 /dev/net/tun
ofirule
  • 4,233
  • 2
  • 26
  • 40
Melardev
  • 1,101
  • 10
  • 22
  • Can you provide full details of the errors in your question? – gohm'c Nov 24 '21 at 00:37
  • @gohm'c My issue is simple, I just did not have internet connection, neither internal nor external, Now I made an improvement, I made external networking work, but not internal. See the Updates please – Melardev Nov 24 '21 at 00:55
  • 1
    Hi, @Melardev . Now that you've made external connection work, I suppose you are at the same point as the person from the linked post. Try to configure two routes: one for Kubernetes service subnet and one for pod subnet. Usually, these are `10.43.0.0/16` and `10.42.0.0/16` respectively, but it is better to check that. For example, using this line `route 10.42.0.0 255.255.0.0 net_gateway` you are telling that to reach any IP in `10.42.0.0/16`, the packet should go via the default gateway and NOT VPN tunnel. In other words, all traffic goes through the VPN except for IPs in `10.42.0.0/16`. – anemyte Nov 24 '21 at 07:16
  • @anemyte Hi, thanks but still not resolved, with kubectl get pod -o wide (is there a better command?) I see the internal IP for the pod is 192.168.87.228, so in my .ovpn file I added `route 192.168.0.0 255.255.0.0 net_gateway`and also your two lines above (10.43.0.0 and 10.42.0.0 just in case), as a test I had a http server on the host on port 3002, trying curl 192.168.1.138 does not work with VPN, without VPN works this is the output for route pre and post: https://gist.github.com/melardev/9a1c9653ac41835625469ed1e5b6b77b – Melardev Nov 24 '21 at 08:55
  • @Melardev please add the `curl` command that you used along with `traceroute` example and I'll look into this later. – anemyte Nov 24 '21 at 09:26
  • @anemyte sorry, actually the general setup works, which is the most important, so I think we can consider this issue resolved, however as I said the curl 192.168.1.138 I could not make it work, not sure if it is expected or not, today I can't even without VPN, yesterday I did (only without VPN)... If you don't have any further suggestion then It is ok, I would consider this resolved as the most important is done. Outputs: https://gist.github.com/melardev/91912a594e3bf13bb40693e7fc2204f8 – Melardev Nov 24 '21 at 09:49
  • @anemyte I found the reason why this worked yesterday, it did not work on K8s it worked on raw docker, adding the OpenVPN route rules made it work (the curl 192.168.1.138 command) but only on `docker run ` not on Kubernetes pod deployment – Melardev Nov 24 '21 at 09:55
  • @Melardev Did I understand it correctly that you've made it working with Docker but not with K8s? If so, would you like me to share a working pod spec for a VPN client? I see a clear difference between pod spec and `docker run` in `tun` device mount. – anemyte Nov 24 '21 at 15:46
  • @anemyte I would greatly appreciate if you take the time to create and provide a working proof of concept, I am new to k8s and networking so although my current status is I have a seemingly working example, it is not well tested, and I feel like the internal networking is broken. A showcase of a correct way of doing it would be perfect. for beginners like me – Melardev Nov 24 '21 at 16:53

2 Answers2

7

Here is a minimal example of a pod with OpenVPN client. I used kylemanna/openvpn as a server and to generate a basic client config. I only added two routes to the generated config to make it working. See below:

apiVersion: v1
kind: Pod
metadata:
  name: ovpn
  namespace: default
spec:
  containers:
    - name: ovpn
      image: debian:buster
      args:
        - bash
        - -c
        # install OpenVPN and curl; use curl in an endless loop to print external IP
        - apt update && apt install -y openvpn curl && cd /config && openvpn client & while sleep 5; do echo $(date; curl --silent ifconfig.me/ip); done
      volumeMounts:
        - mountPath: /dev/net/tun
          readOnly: true
          name: tun-device
        - mountPath: /config
          name: config
      securityContext:
        capabilities:
          add: ["NET_ADMIN"]
  volumes:
    - name: tun-device
      hostPath:
        path: /dev/net/tun
    - name: config
      secret:
        secretName: ovpn-config
---
apiVersion: v1
kind: Secret
metadata:
  name: ovpn-config
  namespace: default
stringData:
  client: |

    # A sample config generated by https://github.com/kylemanna/docker-openvpn server
    client
    nobind
    dev tun

    # Remote server params
    remote PASTE.SERVER.IP.HERE 1194 udp

    # Push all traffic through the VPN
    redirect-gateway def1
    # except these two k8s subnets
    route 10.43.0.0 255.255.0.0 net_gateway
    route 10.42.0.0 255.255.0.0 net_gateway

    # Below goes irrelevant TLS config
    remote-cert-tls server
    <key>
    -----BEGIN PRIVATE KEY-----
    -----END PRIVATE KEY-----
    </key>
    <cert>
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
    </cert>
    <ca>
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
    </ca>
    key-direction 1
    <tls-auth>
    #
    # 2048 bit OpenVPN static key
    #
    -----BEGIN OpenVPN Static key V1-----
    -----END OpenVPN Static key V1-----
    </tls-auth>
anemyte
  • 17,618
  • 1
  • 24
  • 45
  • thanks for the help, this is the solution I was looking for. – Melardev Nov 24 '21 at 20:00
  • I need to redirect my application to a local ingress, but this doesn't work I'm getting error 504. Any idea what I should add? – Reda Drissi Feb 18 '22 at 19:16
  • @RedaDrissi All I can say with this amount of data that it is probably not working :) I suggest you ask a question and bring the details on what you have, what you want to achieve, and what you tried so far. Then we'll see how it can be solved. – anemyte Feb 18 '22 at 19:24
  • @anemyte thanks but I solved it through adding my pod IP range to `LOCAL_NETWORK` not ideal since it adds a security issue, but good enough for my use case. – Reda Drissi Feb 19 '22 at 09:36
  • Hi @Melardev and anemyte I came across this question and it's similar to my problem, link below. Can you please take a look at it, as I've tried almost anything I can think of and can see that you guys have a better understanding of VPN networking than I do. I'd be extremely grateful :) https://stackoverflow.com/questions/72350518/how-to-access-my-app-over-vpn-if-its-deployed-in-a-multicontainer-pod-using-ngin – Erokos May 23 '22 at 14:53
-4

Try Tailscale. https://tailscale.com/ It's much simpler. And they have a cool free-tier

Piero
  • 1,638
  • 1
  • 13
  • 14
  • Thanks for the suggestion, I did not know about, but it is not exactly what I was looking for – Melardev Nov 25 '21 at 09:38
  • I have it running in a k8s cluster. It makes a VPN gateway to the cluster so I see the pods on my local machine as if they were on my local network. It's a 10 mn install and it just ...works. There's an openVPN comparison here : https://tailscale.com/kb/1170/tailscale-vs-openvpn/ – Piero Nov 25 '21 at 12:29