Connecting to Ray Cluster through VirtualService

Question

PROBLEM:

I have a Ray Cluster (version 2.2.0) in Kubernetes, and I would like to connect to it through a VirtualService rather than its internal service address.

When a connection is established by head-service URL ray.init(address="ray://kuberay-head-svc.kuberay:10001") everything is fine. But, when a connection is attempted via VirtualService URL as in ray.init(address="https://ray-cluster.<domain_name>"), then I receive following error;

RuntimeError: Module: https does not exist.
This module was parsed from Address: https://ray-cluster.<domain_name>

My VirtualService is as follows;

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ray-cluster-vs
  namespace: kuberay
spec:
  hosts:
    - "ray-cluster.<domain_name>"
  gateways:
    - istio-system/some-gateway
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: kuberay-head-svc.kuberay.svc.cluster.local
            port:
              number: 10001

WHAT I HAVE DONE:

Initially, I thought it was due to missing python package https in the head pod and tried to install the related package, but it was already installed;

(base) ray@kuberay-head-6626v:~$ pip install pyopenssl
Requirement already satisfied: pyopenssl in ./anaconda3/lib/python3.8/site-packages (22.0.0)
Requirement already satisfied: cryptography>=35.0 in ./anaconda3/lib/python3.8/site-packages (from pyopenssl) (38.0.1)
Requirement already satisfied: cffi>=1.12 in ./anaconda3/lib/python3.8/site-packages (from cryptography>=35.0->pyopenssl) (1.15.1)
Requirement already satisfied: pycparser in ./anaconda3/lib/python3.8/site-packages (from cffi>=1.12->cryptography>=35.0->pyopenssl) (2.21)
(base) ray@kuberay-head-6626v:~$

Somehow, the Cluster expects ray:// protocol in connection URL, and of course when I simply try ray:// protocol in my VirtualService URL, it is not resolved by the platform and causes ConnectionError: ray client connection timeout error.

I have seen similar issues such as https://discuss.ray.io/t/unable-to-connect-to-ray-cluster/7292/25, but not able to find exact same issue online.

Any help would be appreciative.

Thanks a lot!

EDIT: Looks like this is not supported feature in Ray. So, I have created a feature request which is marked as P1 https://github.com/ray-project/ray/issues/35079

Connecting to Ray Cluster through VirtualService

0 Answers0