0

I'm using thorntail microprofile framework to monitor a simple rest service application. The application deployment on openshift works fine but not the health monitor since receive this message:

Readiness probe failed: Get http://10.116.0.57:8080/health/live: dial tcp 10.116.0.57:8080: connect: connection refused

But can access to the health service using the service route url e.g. http://thorntail-myproject.apps-crc.testing/health/live and get the results:

{"status":"UP","checks":[{"name":"server-state","status":"UP"}]}

Both Liveness and Readiness annotations are included in the HealthCheck implementation class. Also get service response when execute curl through pod's remote container shell.

These are the dependencies I'm using in pom.xml:

<dependencies>
  <dependency>
    <groupId>io.thorntail</groupId>
    <artifactId>jaxrs</artifactId>
  </dependency>
  <dependency>
    <groupId>io.thorntail</groupId>
    <artifactId>microprofile-health</artifactId>
  </dependency>
</dependencies>

Any ideas?

iperezmel78
  • 415
  • 5
  • 20
  • I don't know, but did you try to access the URL from inside the pod? You can access the pod's terminal via the OpenShift console, or using `oc rsh`, and then use `curl`. That would be my first debugging step, anyway. – Ladicek Jul 24 '20 at 06:52
  • Already tried that and also get response from service. Question updated. – iperezmel78 Jul 24 '20 at 18:51
  • Are you perhaps using Istio, or something else that messes with container networking? I remember trying to use a simple HTTP probe and failing with Istio -- had to replace it with a command based probe that simply did `curl http://localhost:8080/health/...`. – Ladicek Jul 26 '20 at 20:47
  • Hi @Ladicek, thanks for your replies!. I'm not using isitio, in my pom.xml only have jaxrs and microprofile-health dependencies. I have updated the question. – iperezmel78 Jul 27 '20 at 13:52
  • At this point, I have no idea what could be the problem, but it seems unlikely it would be the application itself. – Ladicek Jul 27 '20 at 17:27

1 Answers1

0

The problem could be caused by many things, but here are some things you can try:

  1. Verify that the Service object for your deployment / deploymentConfig is connecting to the correct Pods and to the correct Ports.
  2. Verify that the Route/Ingress objects are connecting to the correct Service object.

The two things above seem correct as you can access the Route URL, but we don't know your deployments and how many you have.

  1. Verify that your Liveness and Readiness probes are hitting the correct: Ports, page (might be a typo somewhere), protocol - are you using HTTP or HTTPS?
  2. If all of the above is correct, check if you have additional NetworkPolicies for your namespace.
hilchev
  • 340
  • 1
  • 2
  • 9
  • Hi @Hilchev. Only have 2 pods deployed on Openshift. The other one is a mysql and it's working fine. Don't really know what could be the cause of the issue. When doing curl from shell, I do copy the url from readiness probe, paste it and get results. And for your last recommendation, I don't have configured network policies in the namespace. Anyway thanks for the advices!. – iperezmel78 Aug 05 '20 at 05:35
  • In my experience, these issues are caused mainly by two things: 1. Incorrect Service configuration (e.g. the Service object has the wrong port open) 2. The application is not configured correctly. In your case, you can hit the Health page from the shell, which looks like the case is #1. Is your Service hitting the correct pods? Is your Service exposing the correct ports? From the SHELL, did you hit localhost or the Service URL? If it's localhost, it might be that your app doesn't listen on 0.0.0.0 and as such can't receive traffic outside of localhost. – hilchev Aug 13 '20 at 08:06