1

I building a Kubernetes cluster on Google Kubernetes Engine (GKE). It is basically a Service with an associated ReplicaSet with a number of pods.

Those pods need to talk to each other for keeping consensus. For this end, a ClusterIP seems a good fit, allowing intra cluster communication of the pods.

However, now I want to expose this Service to the world. My idea was to switch from ClusterIP to NodePort and couple it with an Ingress, which seems to be the best practice.

My problem is that when I switch the Service to NodePort, I lose the internal communication of the cluster, i.e. pods can't talk to each other. As far as my understanding goes, NodePort is a superset of ClusterIP, so it should keep internal communication.

What am I doing wrong?

Edit with extra information:

I am referring to this example, an example of a Neo4j graph database.

The example deploys a StatefulSet, in which pods need to communicate, among other things, for keeping consensus between the cluster.

With the provided setting, pods can talk to each other. If I change the Service to NodePort, and fix the nodePorts that are used (instead of choosing them randomly as it normally does), pods can no longer communicate.

Is this expected behavior, or am I missing something?

adrpino
  • 960
  • 8
  • 33
  • 1
    That must be a GKE-specific bug, because I am 100% positive NodePort and ClusterIP coexist, since easily 80% of our Pods in our cluster use the ClusterIP to speak to other Services (and a great deal of our Services are NodePort exactly for the Ingress reason you mentioned). Now what I _can't_ say with 100% certainty is whether they can speak to _each other_ that way. – mdaniel Dec 22 '17 at 08:46
  • 1
    Can you expand on what "lose the internal communication" looks like in practice? DNS failure, TCP rejection, TCP timeout, X out of Y connections fail, ... that kind of thing – mdaniel Dec 22 '17 at 08:47
  • I added the example in the question. Losing internal communication means that pods, they stay forever in the following state: `2017-09-13 09:39:59.977+0000 INFO Attempting to connect to the other cluster members before continuing...` – adrpino Dec 22 '17 at 08:58

1 Answers1

1

Indeed NodePort is a superset of ClusterIP, but to be clear, you do not need the service to be of NodePort type for it to be exposed by IngressController. IC has access to endpoints (pods) directly, so there is no need to use anything but ClusterIP.

Another thing is that ClusterIP service has no effect on pod-to-pod connectivity, and also it seems a bit weird that you would use service to allow consensus chatter (unless you have an svc per pod). For this kind of operations you might want to look closer at the StatefulSet concept

Radek 'Goblin' Pieczonka
  • 21,554
  • 7
  • 52
  • 48
  • As far as I know, in GKE, you [can't](https://stackoverflow.com/a/37514073/4766124) expose a `ClusterIP` with an Ingress, but I'm not 100% sure. – adrpino Dec 22 '17 at 09:16