0

Client-side is throwing an error when trying to connect to an Anthos service deployed in an GKE cluster.

WebSocket connection to 'wss://api.beta.tld' failed: Error during WebSocket handshake: Unexpected response code: 503

Client-side can do https request to Anthos service just fine, ie. https://api.beta.tld/ping so it's only the secure websockets that are throwing 503s, I can't use ws:// as browsers blocks all request due to mixed content.

According to GKE docs websockets should be supported out of the box: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#support_for_websocket

Cluster is setup as following:


  gcloud container --project ${GOOGLE_PROJECT_ID} clusters create "${CLUSTER_NAME}" \
    --zone ${GOOGLE_COMPUTE_ZONE} \
    --machine-type "e2-small" \
    --disk-type "pd-ssd" \
    --disk-size "10" \
    --preemptible --num-nodes "3" \
    --addons HorizontalPodAutoscaling,HttpLoadBalancing,NodeLocalDNS \
    --enable-autoscaling --min-nodes "1" --max-nodes "9" \
    --release-channel "regular" \
    --enable-ip-alias \
    --enable-autoupgrade \
    --enable-autorepair \
    --max-surge-upgrade 1 \
    --max-unavailable-upgrade 0 \
    --enable-shielded-nodes \
    --shielded-secure-boot

  gcloud container clusters update "${CLUSTER_NAME}" --zone=${GOOGLE_COMPUTE_ZONE} --update-addons=CloudRun=ENABLED,HttpLoadBalancing=ENABLED

  gcloud container clusters get-credentials "${CLUSTER_NAME}" --region ${GOOGLE_COMPUTE_ZONE}

  kubectl patch cm config-domainmapping -n knative-serving -p '{"data":{"autoTLS":"Enabled"}}'

And I am deploying services like so:

    gcloud run deploy \
      --project ${GOOGLE_PROJECT_ID} \
      --platform gke \
      --cluster ${CLUSTER_NAME} \
      --cluster-location ${GOOGLE_COMPUTE_ZONE} \
      --image "gcr.io/${GOOGLE_PROJECT_ID}/${PROJECT_NAME}/${BRANCH}" \
      --set-env-vars "$(tr '\n' ',' < ${ENV_KEY_PRODUCTION})" \
      --min-instances 1 \
      --max-instances 20 \
      "${1}"

I've noticed that the cluster is created with a TCP LoadBlanacer not a https, not sure if that's the issue or how to create cluster with HTTPs load balancer and still work with Anthos.

Labithiotis
  • 3,519
  • 7
  • 27
  • 47
  • Issue could be related to load balancer being TCP instead of HTTP(s) as per GCP documentation[1], Looks websocket is supported for HTTP(S) Load balancer. Can you follow documentation[2} and create HTTPS load balancer and see if that mitigates the issue. [1]https://cloud.google.com/load-balancing/docs/https#websocket_support [2]https://cloud.google.com/solutions/integrating-https-load-balancing-with-istio-and-cloud-run-for-anthos-deployed-on-gke – Dattu Pragnu Nellutla Sep 22 '20 at 00:14
  • @DattuPragnuNellutla I had already tried to follow that google tutorial, but in Cloud Run the domain mapping got stuck saying `Waiting for gateway IP`, looks like a HTTP(s) LB was added and updated DNS records and remove domain mappings from cloud run but couldn't connect to service, looks like something is missing with Istio setup – Labithiotis Sep 22 '20 at 08:00
  • Thank you for clarifying. My previous comment regarding support for Websockets for TCP load balancer might be irrelevant as TCP load balancer is pass-through[1 and seems like documentation[2] will setup a TCP load balancer for Authorizing access to Cloud Run for Anthos services using Istio]. Since, our TCP LB is passthrough i think websocket getting 503 could be an issue with backend as every traffic will be automatically sent to backend and the response is directly from backend. I would recommend verifying your backend application. – Dattu Pragnu Nellutla Sep 23 '20 at 01:04
  • [1]https://cloud.google.com/load-balancing/docs/network [2]https://cloud.google.com/solutions/authorizing-access-to-cloud-run-on-gke-services-using-istio – Dattu Pragnu Nellutla Sep 23 '20 at 01:05

0 Answers0