0

I have the following setup: k3s installation and istio as ingress controller,

problem we face that rabbitmq service is not reachable from within pods internally.

  • A Service that has rabbitmq port exposed [5671] -> type: ClusterIP

     $ kubectl describe service rabbit-svc -n rabbit
 ......
 Type:              ClusterIP
 IP Family Policy:  SingleStack
 IP Families:       IPv4
 IP:                10.43.139.126
 IPs:               10.43.139.126
 ......
 Port:              https  443/TCP
 TargetPort:        443/TCP
 Endpoints:         10.42.0.222:443
 Port:              rabbitmq  5671/TCP
 TargetPort:        5671/TCP
 Endpoints:         10.42.0.222:5671

  • if I try to connect to that service via clusterIP from inside the machine outside k3s pods, it responds back

    openssl s_client -connect 10.43.139.126:5671
CONNECTED(00000003)

  • if I try to connect to the pod from istio pod via the service k8s fqdn, it gives connection refused

    openssl s_client -connect rabbit-svc.rabbit.svc.cluster.local:5671
139693269235008:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:110:
139693269235008:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
connect:errno=111

Any idea why this is happening though I have another port from inside that rabbitmq service exposed via 443 and it respond back fine [ that's just to rule out DNS issues ]

AhMaD AbUIeSa
  • 805
  • 1
  • 12
  • 21

1 Answers1

0

The connection error:111 is might be due to below concerns and can you check on this :

  • Check whether the Port 5671 is open or blocked by using nmap or telnet.
  • Check if there are any firewalls blocking this port/host. Try to disable the firewall and have a check
  • Seems to be your istio is having SSL certificate enabled - this blocking the port 5671 to reach the connection. Can you disable the SSL certificate and have a check. If it is connected then SSL certificate is the reason which is blocking this. In this case you need to make sure 5671 Port should accept the SSL certificate. If there is no SSL certificate enabled then the first two points might be the reason.
  • Can you try by using this command echo | openssl s_client -connect <hostname>:<port> | head

Refer to this RabbitMQ troubleshooting doc for more information

Hemanth Kumar
  • 2,728
  • 1
  • 4
  • 19
  • well based on my tests the port is listening and like I mentioned in the question , it can be reached outside the pods, and if you don't pass in the certificate at least the connection would be opened , I also tried from different pods , not just the istio pod – AhMaD AbUIeSa Apr 15 '23 at 10:39
  • As you are using FQDN ,can try you adding [istio Virtualservice](https://codeburst.io/istio-by-example-5189edd043da) and [Destination Rule](https://istio.io/latest/docs/reference/config/networking/destination-rule/#LoadBalancerSettings) to route the traffic to the defined service. – Hemanth Kumar Apr 15 '23 at 11:14
  • well , I added the virtual service along with the gateway , and since that didn't work , I decided to check why, and came to this problem that ingress gateway pod can't establish the connection to the rabbitmq – AhMaD AbUIeSa Apr 15 '23 at 11:49
  • @AhMaDAbUIeSa : Even i'm checking on this but mean while can you check whether you have given the correct format of FQDN hostname as `..svc.cluster.local`. You can check this [SO](https://stackoverflow.com/a/56106626/19230181). – Hemanth Kumar Apr 15 '23 at 12:34
  • but I can see your service name as `rabbit-svc` and name space as `rabbit`. So, try the following : `openssl s_client -connect rabbit-svc.rabbit.svc.cluster.local:5671` let me know if you find any errors. – Hemanth Kumar Apr 15 '23 at 12:43
  • yes you are right , that was just a typo , from different trials , and yes I am using the correct name [ as I said in the question , I can connect to port 443 , which is another port that exposes an https process ] – AhMaD AbUIeSa Apr 15 '23 at 12:49
  • Can you check this [link1](https://community.freepbx.org/t/unable-to-load-ssl-certificate-for-port-5161-but-port-443-works/72570/2) and [Link2](https://forum.howtoforge.com/threads/apache2-openssl-s_client-error.85910/) – Hemanth Kumar Apr 15 '23 at 13:11
  • @AhMaDAbUIeSa : Please let me know whether the shared info was helpful and the issue resolved?. I am happy to assist if you have any further queries. – Hemanth Kumar Apr 17 '23 at 03:58