3

I installed a Spark on K8s operator in my K8s cluster and I have an app running within the k8s cluster. I'd like to enable this app to talk to the sparkapplication CRD service. Can I know what would be the endpoint I should use? (or what's the K8s endpoint within a K8s cluster)

Rico
  • 58,485
  • 12
  • 111
  • 141
injoy
  • 3,993
  • 10
  • 40
  • 69

1 Answers1

1

It's clearly documented here. So basically, it creates a NodePort type of service. It also specifies that it could create an Ingress to access the UI. For example:

...
status:
  sparkApplicationId: spark-5f4ba921c85ff3f1cb04bef324f9154c9
  applicationState:
    state: COMPLETED
  completionTime: 2018-02-20T23:33:55Z
  driverInfo:
    podName: spark-pi-83ba921c85ff3f1cb04bef324f9154c9-driver
    webUIAddress: 35.192.234.248:31064
    webUIPort: 31064
    webUIServiceName: spark-pi-2402118027-ui-svc
    webUIIngressName: spark-pi-ui-ingress
    webUIIngressAddress: spark-pi.ingress.cluster.com

In this case, you could use 35.192.234.248:31064 to access your UI. Internally within the K8s cluster, you could use spark-pi-2402118027-ui-svc.<namespace>.svc.cluster.local or simply spark-pi-2402118027-ui-svc if you are within the same namespace.

Rico
  • 58,485
  • 12
  • 111
  • 141
  • Sorry, @Rico , I didn't make it clear. What I want to implement is a spark app manager. Whenever it receives a message/signal, it will launch an additional spark app into the k8s cluster. The spark app manager stays in the k8s cluster as well (same namespace as the spark apps). So the manager should be able to talk to the k8s sparkapplication API. So how can I config the API client? (what's the base URL and authentications...) – injoy Feb 07 '19 at 18:45