How can I access the Openshift service through ClusterIP from nodes

Question

I am trying to access a Flask server running in one Openshift pod from other.

For that I created a service as below.

$ oc get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
my-web-app   ClusterIP   172.30.216.112   <none>        8080/TCP   8m


$ oc describe svc my-web-app
Name:              my-web-app
Namespace:         neo4j-sys
Labels:            app=my-web-app
Annotations:       openshift.io/generated-by=OpenShiftNewApp
Selector:          app=my-web-app,deploymentconfig=my-web-app
Type:              ClusterIP
IP:                172.30.216.112
Port:              8080-tcp  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.20.203.104:5000,172.20.49.150:5000
Session Affinity:  None
Events:            <none>

1) First, I ping ed from one pod to other pod and got response.

(app-root) sh-4.2$ ping 172.20.203.104
PING 172.20.203.104 (172.20.203.104) 56(84) bytes of data.
64 bytes from 172.20.203.104: icmp_seq=1 ttl=64 time=5.53 ms
64 bytes from 172.20.203.104: icmp_seq=2 ttl=64 time=0.527 ms
64 bytes from 172.20.203.104: icmp_seq=3 ttl=64 time=3.10 ms
64 bytes from 172.20.203.104: icmp_seq=4 ttl=64 time=2.12 ms
64 bytes from 172.20.203.104: icmp_seq=5 ttl=64 time=0.784 ms
64 bytes from 172.20.203.104: icmp_seq=6 ttl=64 time=6.81 ms
64 bytes from 172.20.203.104: icmp_seq=7 ttl=64 time=18.2 ms
^C
--- 172.20.203.104 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6012ms
rtt min/avg/max/mdev = 0.527/5.303/18.235/5.704 ms

But, when I tried curl, it is not responding.

(app-root) sh-4.2$ curl 172.20.203.104
curl: (7) Failed connect to 172.20.203.104:80; Connection refused

(app-root) sh-4.2$ curl 172.20.203.104:8080
curl: (7) Failed connect to 172.20.203.104:8080; Connection refused

2) After that I tried to reach cluster IP from one pod. In this case, both ping, curl not reachable.

(app-root) sh-4.2$ ping 172.30.216.112
PING 172.30.216.112 (172.30.216.112) 56(84) bytes of data.
From 172.20.49.1 icmp_seq=1 Destination Host Unreachable
From 172.20.49.1 icmp_seq=4 Destination Host Unreachable
From 172.20.49.1 icmp_seq=2 Destination Host Unreachable
From 172.20.49.1 icmp_seq=3 Destination Host Unreachable
^C
--- 172.30.216.112 ping statistics ---
7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6002ms
pipe 4
(app-root) sh-4.2$ curl 172.30.216.112
curl: (7) Failed connect to 172.30.216.112:80; No route to host

Please let me know where I am going wrong here. Why the above cases #1, #2 failing. How to access the clusterIP services.

I am completely new to the services and accessing them and hence I might be missing some basics.

I gone through other answers How can I access the Kubernetes service through ClusterIP 1. But, it is for Nodeport which is not helping me.

Updates based on below comment from Graham Dumpleton, below are my observations.

This is the Flask server log which I am running in the pods for information.

* Serving Flask app "wsgi" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/  (Press CTRL+C to quit)
127.0.0.1 - - [14/Nov/2019 04:54:53] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [14/Nov/2019 04:55:05] "GET / HTTP/1.1" 200 -

Is your pod listening on external interfaces on pod 8080?

If I understand question correctly, my intention here is just to communicate between pods via clusterIP service. I am not looking for accessing pods from external interfaces(other projects or through web urls as load balancer service)

If you get into the pod, can you do curl $HOSTNAME:8080?

Yes, if I am running as localhost or 127.0.0.1, I am getting response from the same pod where I run this as expected.

(app-root) sh-4.2$ curl http://127.0.0.1:5000/  
Hello World!

(app-root) sh-4.2$ curl http://localhost:5000/  
Hello World!

But, if I tried with my-web-app or service IP(clusterIP). I am not getting response.

(app-root) sh-4.2$ curl http://172.30.216.112:5000/
curl: (7) Failed connect to 172.30.216.112:5000; No route to host

(app-root) sh-4.2$ curl my-web-app:8080
curl: (7) Failed connect to my-web-app:8080; Connection refused

(app-root) sh-4.2$ curl http://my-web-app:8080/
curl: (7) Failed connect to my-web-app:8080; Connection refused

With pod IP also I am not getting response.

(app-root) sh-4.2$ curl http://172.20.49.150:5000/
curl: (7) Failed connect to 172.20.49.150:5000; Connection refused

(app-root) sh-4.2$ curl 172.20.49.150
curl: (7) Failed connect to 172.20.49.150:80; Connection refused

Is your pod listening on external interfaces on pod 8080? If you get into the pod, can you do `curl $HOSTNAME:8080`? From other pods, to access it via the service, use `my-web-app:8080`, don't use the IP. This will only work if in same project. If in separate project and OpenShift has multitenant network overlay used, you need to open up access between projects. — Graham Dumpleton, Nov 13 '19 at 23:13
@Graham Dumpleton. Updated the answer with observations. Please go through it. — Praveen L, Nov 14 '19 at 05:34
When I said to use `$HOSTNAME:8080` I really meant that. The `HOSTNAME` environment variable is the pod IP and behaves differently to using `127.0.0.0` or `localhost`. Anyway, your log says that you are only listening on local interface, so that shouldn't work and you can't connect via pod IP or from outside. You must get Flask to bind to host `0.0.0.0` and not `127.0.0.1` or `localhost`. — Graham Dumpleton, Nov 14 '19 at 12:03
I will check for options to bind Flask to `0.0.0.0`. But, if that happens, please let me know how it can solve my issue. — Praveen L, Nov 14 '19 at 12:08
I bind the Flask server to `0.0.0.0` and now I am able to `curl` successfully either by `curl http://0.0.0.0:5000/ ` or `curl $HOSTNAME:5000` or with clusterIP `curl 172.30.216.112:8080`. Thanks for the info. — Praveen L, Nov 14 '19 at 12:48
Does it solved your problem completely or do you still have issue with something? — PjoterS, Nov 14 '19 at 14:40
It solved my problem now. I am able to communicate with the Flask server via clusterIP service after I changed the binding to `0.0.0.0` as suggested by @Graham Dumpleton — Praveen L, Nov 15 '19 at 05:19

score 0 · Answer 1 · answered Nov 15 '19 at 05:42

I am answering my own question. Here is how my issue got resolved based on inputs from Graham Dumpleton.

Initially, I used to connect Flask server as below.

from flask import Flask
application = Flask(__name__)

if __name__ == "__main__":
    application.run()

This bind the server to http://127.0.0.1:5000/ by default.

As part of resolution I changed the bind to 0.0.0.0 as below

from flask import Flask
application = Flask(__name__)

if __name__ == "__main__":
    application.run(host='0.0.0.0')

And log as below after that.

* Serving Flask app "wsgi" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/  (Press CTRL+C to quit)

After that pods successfully communicated via clusterIP. Below are the service details(increased one more pod)

$ oc describe svc my-web-app
Name:              my-web-app
Namespace:         neo4j-sys
Labels:            app=my-web-app
Annotations:       openshift.io/generated-by=OpenShiftNewApp
Selector:          app=my-web-app,deploymentconfig=my-web-app
Type:              ClusterIP
IP:                172.30.4.250
Port:              8080-tcp  8080/TCP
TargetPort:        5000/TCP
Endpoints:         172.20.106.184:5000,172.20.182.118:5000,172.20.83.40:5000
Session Affinity:  None
Events:            <none>

Below is the successful response.

(app-root) sh-4.2$ curl http://172.30.4.250:8080  //with clusterIP which is my expectation
Hello World!

(app-root) sh-4.2$ curl http://172.20.106.184:5000 // with pod IP
Hello World!

(app-root) sh-4.2$ curl $HOSTNAME:5000    // with $HOSTNAME
Hello World!

It is strongly discouraged to use the clusterIP for a service. Use the name of the service object, as a hostname. That way you don't need to know the IP, which will change for each deployment. So use `http://my-web-app:8080` when communicating with it from other applications in the same project. — Graham Dumpleton, Nov 15 '19 at 20:06
Thanks for suggestion. I am able to connect as `http://my-web-app:8080` and I will use this form only to communicate. — Praveen L, Nov 18 '19 at 09:32

How can I access the Openshift service through ClusterIP from nodes

1 Answers1