3

Relates to How to call a service exposed by a Kubernetes cluster from another Kubernetes cluster in same project.

Asking again since Kubernetes has been changes a lot since July.

Context:

I'm working on an infrastructure with multiple clusters serving different purposes, e.g.:

  • Cluster A runs services/apps creating data for consumption
  • Cluster B runs services/apps consuming data created by apps in cluster A
  • Cluster C runs data services like Redis, Memcache, etc.

All clusters are in the default namespace.

Problem:

In Kubernetes, each cluster gets its own kubernetes (in the default namespace) and kube-dns (in the kube-system namespace) service with a different IP.

What happens with this setup is that, services in cluster A and B above can't discover (in service discovery terminology), let's say, Redis in cluster C. So a nslookup redis.default.svc.cluster.local from one of the services in cluster A/B comes back with ** server can't find redis.default.svc.cluster.local: NXDOMAIN. Note: This works from within cluster C.

I've read as many documents as I found about kube-dns, and pretty much all assume one cluster setup.

Clusters info:

Here are /etc/resolv.conf from two different clusters showing DNS nameservers with no common kube-dns ancestor:

Cluster A:

nameserver 10.67.240.10
nameserver 169.254.169.254
nameserver 10.240.0.1
search default.svc.cluster.local svc.cluster.local cluster.local c.project-name.internal. 1025230764914.google.internal. google.internal.

Cluster C:

nameserver 10.91.240.10
nameserver 169.254.169.254
nameserver 10.240.0.1
search default.svc.cluster.local svc.cluster.local cluster.local c.project-name.internal. google.internal.
options ndots:5

Both clusters have these services running with their respective IPs for their cluster in the kube-system namespace:

NAME                  LABELS                                                                           SELECTOR           
kube-dns              k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS   k8s-app=kube-dns
kube-ui               k8s-app=kube-ui,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeUI     k8s-app=kube-ui
monitoring-heapster   kubernetes.io/cluster-service=true,kubernetes.io/name=Heapster                   k8s-app=heapster

What is the ideal fix/update to this setup that can get the shared services discovered across all Kubernetes clusters in a GCE environment?

Community
  • 1
  • 1
mohsenrezaeithe
  • 85
  • 1
  • 8

1 Answers1

5

This is one of the large problems that Kubernetes is trying to solve with Cross-Cluster Service Discovery as a part of the Cluster Federation plans. You can also check out/contribute to the Federation SIG.

If you've used one of the hacks solutions described here, you might be able to hack up your /etc/resolve.conf to also search the nameserver from the other cluster. Be careful, because this may run you into truncation issues.

You might also be able to modify the sky-dns RC for your clusters to include an extra kube2sky pod that points at the other cluster's kubernetes service (I haven't tried this, or thought through all of the implications).

Neither of the two hacks I've described above would prevent name collision, so you'd have to manually prevent that.

Community
  • 1
  • 1
CJ Cullen
  • 5,452
  • 1
  • 26
  • 34
  • Thanks for information, CJ! I've ran into these hacks/solutions, and with Kube Federation being around half a year away, I've settled for single cluster solution at this point, until Kubernetes has a proper solution. – mohsenrezaeithe Nov 24 '15 at 22:52