Pod to Pod communication within a Service

Question

Is it possible to send a http Rest request to another K8 Pod that belongs to the same Service in Kubernetes?

E. G. Service name = UserService , 2 Pods (replica = 2)

Pod 1 --> Pod 2 //using pod ip not load balanced hostname 
Pod 2 --> Pod 1

The connection is over Rest GET 1.2.3.4:7079/user/1

The value for host + port is taken from kubectl get ep

Both of the pod IP's work successfully outside of the pods but when I do a kubectl exec -it into the pod and make the request via CURL, it returns a 404 not found for the endpoint.

Q What I would like to know if it is possible to make a request to another K8 Pod that is in the same Service?

Q Why am I able to get a successful ping 1.2.3.4, but not hit the Rest API?

below is my config files

 #values.yml
replicaCount: 1

 image:
  repository: "docker.hosted/app"
  tag: "0.1.0"
  pullPolicy: Always
  pullSecret: "a_secret"

service:
 name: http
 type: NodePort
 externalPort: 7079
 internalPort: 7079

ingress:
 enabled: false

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "app.fullname" . }}
  labels:
    app: {{ template "app.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "app.name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:

            - name: MY_POD_IP
              valueFrom:
               fieldRef:
                fieldPath: status.podIP
            - name: MY_POD_PORT
              value: "{{ .Values.service.internalPort }}"
          ports:
            - containerPort: {{ .Values.service.internalPort }}
          livenessProbe:
            httpGet:
              path: /actuator/alive
              port: {{ .Values.service.internalPort }}
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/ready
              port: {{ .Values.service.internalPort }}
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- if .Values.nodeSelector }}
      nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
    {{- end }}
      imagePullSecrets:
        - name: {{ .Values.image.pullSecret }

service.yml

kind: Service
metadata:
  name: {{ template "app.fullname" . }}
  labels:
    app: {{ template "app.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.externalPort }}
      targetPort: {{ .Values.service.internalPort }}
      protocol: TCP
      name: {{ .Values.service.name }}
  selector:
    app: {{ template "app.name" . }}
    release: {{ .Release.Name }}

executed from master

executed from inside a pod of the same MicroService

Fascinating, well, based solely on that output, it seems like there is something weird going on with your envoy proxy. I have zero experience with envoy, so maybe ask your question again being envoy specific -- since the question about pod-to-pod communication isn't accurate. Your question is actually pod-to-envoy which is an entirely different ball of wax — mdaniel, Jan 30 '19 at 06:34
OK thanks Matthew, I have asked the question with Envoy tags here https://stackoverflow.com/q/54437129/694960, but I am not hopeful of a response — M_K, Jan 30 '19 at 10:31

score 1 · Answer 1 · answered Jan 29 '19 at 07:35

1

Is it possible to send a http Rest request to another K8 Pod that belongs to the same Service in Kubernetes?

For sure, yes, that's actually exactly why every Pod in the cluster has a cluster-wide routable address. You can programmatically ask kubernetes for the list of the Pod's "peers" by requesting the Endpoint object that is named the same as the Service, then subtract out your own Pod's IP address. It seems like you kind of knew that from kubectl get ep, but then you asked the question, so I thought I would be explicit that your experience wasn't an accident.

Q Why am I able to get a successful ping 1.2.3.4, but not hit the Rest API?

We can't help you troubleshoot your app without some app logs, but the fact that you got a 404 and not "connection refused" or 504 or such means your connectivity worked fine, it's just the app that is broken.

answered Jan 29 '19 at 07:35

mdaniel

31,240
5
55
58

Thanks Matthew, I will check the app logs again, but I can successfully send a Rest request to each Pod with the exact same URL only on the master host, which is leading to believe it is a config issue with ip or port on the K8 yaml side, do you know why this would work on the outside but not inside the first POD? – M_K Jan 29 '19 at 08:52
Then please update your question with the output of `curl -v` from the master, and then from within the cluster, because we can't guess what the reason is. It sounds a little like your SDN might not be working, but hard to know for sure – mdaniel Jan 29 '19 at 17:01
Hi @Matthew, I have added the output from the CURL request, hopefully you can help – M_K Jan 29 '19 at 21:08
Hi @Matthew, thanks can you share what exactly the Pod to Pod communication is named ? I would like to read the exact documentation to understand it more – M_K Feb 01 '19 at 09:20

score 0 · Accepted Answer · edited Feb 20 '19 at 15:16

Yes as Mathew answered you can indeed communicate between pods in a Service with Kubernetes, the problem I was having was Istio was blocking the requests to each other.

Solution: Disabling Istio injection solved this problem for me , I then enabled it after wards and load balancing continued, hopefully it can help some one in the future.

see answer here: Envoy Pod to Pod communication within a Service in K8

Pod to Pod communication within a Service

deployment.yml

service.yml

executed from master

executed from inside a pod of the same MicroService

2 Answers2

Linked