Failed to connect to port 8080: Connection timed out

Question

I'm trying to deploy bazel-remote, but when I try to:

curl http://bazel-remote.dev.azr.myhost.com:80/status

into the ingress host, I'm getting:

<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

back. For reference, I've tried:

curl http://bazel-remote.dev.azr.myhost.com:8080/status

but it gave the same results as when I specified the port number as any arbitrary port, and timed out.

I checked the status of each of the pods, services, ingresses on lens, and they all seem to be running fine. Similar story for the volumes. I've deployed them through Terraform, and they all seem to be operating fine.

Strange things I don't get:

The pods enter CrashLoop when I try specify BAZEL_REMOTE_HOST in the ConfigMap. Most likely due to this.
http://bazel-remote.dev.azr.myhost.com:8080/metrics is accessible via my web browser, detailing the bazel-remote caching capabilities.
When I specified in .bazelrc : build --remote_cache=http://bazel-remote.dev.azr.myhost.com:8080, it didn't work, but when I took the pod IP addresses directly, and put that instead, the remote caching worked as intended.

Suspects: I suspect there is something wrong with my ingress, since the pods are definitely up, and functioning if I target them directly.

Ingress:

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: infra--bazel-remote-cache
  name: bazel-remote-cache
  annotations:
    kubernetes.io/ingress.class: nginx-internal
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    cert-manager.io/cluster-issuer: selfsigned-cluster-issuer
spec:
  tls:
  - hosts:
    - bazel-remote.dev.azr.myhost.com
    secretName: tls-secret
  rules:
  - host: bazel-remote.dev.azr.myhost.com:
    http:
      paths:
      - backend:
          serviceName: bazel-remote-service
          servicePort: 8080
        path: /(.*)

Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: bazel-remote-cache-config
  namespace: infra--bazel-remote-cache
data:
  BAZEL_REMOTE_DIR: /mnt/bazel_remote_backend
  BAZEL_REMOTE_MAX_SIZE: "10"
  BAZEL_REMOTE_GRPC_PORT: "9092"
  BAZEL_REMOTE_PORT: "8080"
  BAZEL_REMOTE_ENABLE_ENDPOINT_METRICS: "true"
  BAZEL_REMOTE_EXPERIMENTAL_REMOTE_ASSET_API: "true"

Deployment & Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bazel-remote-cache
  namespace: infra--bazel-remote-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bazel-remote-cache
  template:
    metadata:
      labels:
        app: bazel-remote-cache
    spec:
      containers:
      - name: bazel-remote-cache
        image: buchgr/bazel-remote-cache:v2.1.1
        resources:
          limits:
            memory: 256Mi
            cpu: "200m"
          requests:
            memory: 256Mi
            cpu: "200m"
        volumeMounts:
        - mountPath: "/mnt/bazel_remote_backend"
          name: bazel-remote-cache
        ports:
        - containerPort: 8080
          protocol: TCP
        envFrom:
        - configMapRef:
            name: bazel-remote-cache-config
      volumes:
      - name: bazel-remote-cache
        persistentVolumeClaim:
          claimName: pvc-bazel-remote

---
apiVersion: v1
kind: Service
metadata:
  name: bazel-remote-service
  namespace: infra--bazel-remote-cache
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: bazel-remote-cache

I am trying to replicate the issue but I'm getting this error when I am trying to access application via hosted website (ingress) or just running `curl` command to call the service - `resource name must be a SHA256 hash in hex. got '/'`. If I tried another path, like `/status`, I am getting: `resource name must be a SHA256 hash in hex. got '/status` etc. Did you get following error? Could you check if the image is correct? — Mikolaj S., Aug 23 '21 at 15:12
@MikolajS. I did not get the error with either. And the image is definitely correct, since the remote cache worked when the pod IP was specified. — M80, Aug 24 '21 at 18:01

Mikolaj S. · Accepted Answer · 2021-08-27T15:07:18.210

NGINX Ingress Controller is only working using HTTP or HTTPS:

An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

So you can't use a custom port 8080 - it won't work.

Based on NGINX Ingress Controller docs:

By default the controller redirects HTTP clients to the HTTPS port 443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress.

This can be disabled globally using ssl-redirect: "false" in the NGINX config map, or per-Ingress with the nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation in the particular resource.

You have setup TLS, so Ingress Controller is redirecting you to HTTPS port.

Change your curl command to:

curl https://bazel-remote.dev.azr.myhost.com/status

It should work.

Failed to connect to port 8080: Connection timed out

1 Answers1

Linked