k3d multi cluster communication

Question

Assuming having 2 separate k3d clusters (namely: vault, dev) is there is a way to have a distinct URL for each cluster (preferably with https) for example: vault.cluster.internal and dev.cluster.internal and allow apps deployed in dev.cluster.internal to lookup something or interact with apps in the vault.cluster.internal ?

The cluster definitions are as follows: dev.yaml:

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: dev
servers: 1
agents: 3
network: k3d-cluster
kubeAPI:
  host: "dev.cluster.internal"
  hostIP: "127.0.0.1"
image: rancher/k3s:v1.24.3-k3s1
ports:
  - port: 3000:3000
    nodeFilters:
      - loadbalancer
options:
  k3d:
    wait: true
    timeout: "60s"
  k3s:
    extraArgs:
      - arg: --tls-san=dev.cluster.internal
        nodeFilters:
          - server:*
      - arg: --disable=metrics-server
        nodeFilters:
          - server:*
      - arg: --disable=traefik
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: false

and the vault.yaml:

apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: vault
servers: 1
agents: 3
network: k3d-cluster
kubeAPI:
  host: "vault.cluster.internal"
  hostIP: "127.0.0.1"
image: rancher/k3s:v1.24.3-k3s1
ports:
 - port: 8200:8200
    nodeFilters:
      - loadbalancer
options:
  k3d:
    wait: true
    timeout: "60s"
  k3s:
    extraArgs:
      - arg: --tls-san=vault.cluster.internal
        nodeFilters:
          - server:*
      - arg: --disable=metrics-server
        nodeFilters:
          - server:*
      - arg: --disable=traefik
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: false

Can this be done without using service mesh?
Can I update the coredns in the clusters to allow resolving the other cluster host names, and how?
Can this be done with docker network configurations, and how?

This is basically to simulate real world clusters (but for local development)

both clusters are using the same docker network (named: `k3d-cluster`) and when exposing any (ingress) service on any of the clusters, it can be accessed correctly from the host (macos) but not from the other cluster, the `vault.cluster.internal` isn't resolved in the `dev.cluster.internal` — a14m, Aug 29 '22 at 13:39

a14m · Answer 1 · 2022-09-06T22:54:42.407

I found 3 solutions for the problem.

The first solution is to add HostAliases section to the dev cluster definition, and make it point to the external IP of the vault cluster loadbalancer:

for example: you can run the following command on the vault cluster after initializing it

$ kubectl --context k3d-vault --namespace vault get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     ...
...
vault        LoadBalancer   10.43.34.131   172.24.0.3      ...
                                           ^^^^^^^^^^
...

dev.yaml would be

#...
ports:
  - port: 3000:3000
    nodeFilters:
      - loadbalancer
hostAliases:
  - ip: 172.24.0.3
    hostnames:
      - vault.cluster.internal
#...

# (alternatively, this can be automated using the following command without editing `dev.yaml` file)
$ KMS_IP=$(kubectl --context k3d-vault --namespace vault get services | grep LoadBalancer | awk -F " " '{ print $4 }')
$ k3d cluster create --config dev.yaml --host-alias $KMS_IP:vault.cluster.internal

this solution allow resolving of hostname (as you would expect in a production cluster)...

The second solution works similarly but using docker network inspect k3d-cluster (where k3d-cluster is the docker network name in cluster definition)

Similarly, run docker network inspect k3d-cluster and note down the IP of the loadbalancer subnet defined by docker:

...
"cad3f3XXXXXX": {
    "Name": "k3d-vault-serverlb",
    "EndpointID": "47d5XXXX"
    "MacAddress": "02:42:ac:18:00:04",
    "IPv4Address": "172.24.0.4/16",  #<<< This IP can be used in dev cluster HostAliases
    "IPv6Address": ""
}
...

The last solution is simpler but less flexible. it uses host.k3d.internal as the name for the other cluster (allowing to resolve it) but you have to take care of port mapping as all of the clusters would be resolving to use the same URL for the services (which isn't ideal, but easy enough to test multi-cluster communication/bugs/etc). In other words, configure the dev cluster VAULT_ADDR to be host.k3d.internal:8200 instead of vault.cluster.internal:8200 This is not flexible with TLS/HTTPS (AFAIK).

k3d multi cluster communication

1 Answers1