2

I'm trying a simple ingress in gke. Following the example from https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress

the pods are up and running, services are active. When I create ingress I'm getting 

Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Normal   ADD     48m                   loadbalancer-controller  default/my-ingress
  Warning  Sync    2m32s (x25 over 48m)  loadbalancer-controller  Error during sync: Error running backend syncing routine: googleapi: got HTTP response code 404 with body: Not Found

I can't find the source of the problem. Any suggestion of where to look?

I have checked cluster add-ons and permissions httpLoadBalancing enabled

  - https://www.googleapis.com/auth/compute
  - https://www.googleapis.com/auth/devstorage.read_only
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/monitoring
  - https://www.googleapis.com/auth/servicecontrol
  - https://www.googleapis.com/auth/service.management.readonly
  - https://www.googleapis.com/auth/trace.append

NAME                                          READY   STATUS    RESTARTS   AGE
hello-kubernetes-deployment-f6cb6cf4f-kszd9   1/1     Running   0          1h
hello-kubernetes-deployment-f6cb6cf4f-lw49t   1/1     Running   0          1h
hello-kubernetes-deployment-f6cb6cf4f-qqgxs   1/1     Running   0          1h
hello-world-deployment-5cfbc486f-4c2bm        1/1     Running   0          1h
hello-world-deployment-5cfbc486f-dmcqf        1/1     Running   0          1h
hello-world-deployment-5cfbc486f-rnpcc        1/1     Running   0          1h

Name:                     hello-world
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-world","namespace":"default"},"spec":{"ports":[{"port":6000...
Selector:                 department=world,greeting=hello
Type:                     NodePort
IP:                       10.59.254.88
Port:                     <unset>  60000/TCP
TargetPort:               50000/TCP
NodePort:                 <unset>  30418/TCP
Endpoints:                10.56.2.7:50000,10.56.3.6:50000,10.56.6.4:50000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Name:                     hello-kubernetes
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-kubernetes","namespace":"default"},"spec":{"ports":[{"port"...
Selector:                 department=kubernetes,greeting=hello
Type:                     NodePort
IP:                       10.59.251.189
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  32464/TCP
Endpoints:                10.56.2.6:8080,10.56.6.3:8080,10.56.8.6:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Name:             my-ingress
Namespace:        default
Address:
Default backend:  default-http-backend:80 (10.56.0.9:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /*      hello-world:60000 (<none>)
        /kube   hello-kubernetes:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"gce"},"name":"my-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"hello-world","servicePort":60000},"path":"/*"},{"backend":{"serviceName":"hello-kubernetes","servicePort":80},"path":"/kube"}]}}]}}

  kubernetes.io/ingress.class:  gce
Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Normal   ADD     107s                 loadbalancer-controller  default/my-ingress
  Warning  Sync    66s (x15 over 107s)  loadbalancer-controller  Error during sync: Error running backend syncing routine: googleapi: got HTTP response code 404 with body: Not Found

Pulumi Cluster Config

                {
                    "name": "test-cluster",
                    "region": "europe-west4",
                    "addonsConfig": {
                        "httpLoadBalancing": {
                            "disabled": false
                        },
                        "kubernetesDashboard": {
                            "disabled": false
                        }
                    },
                    "ipAllocationPolicy": {},
                    "pools": [
                        {
                            "name": "default-pool",
                            "initialNodeCount": 1,
                            "nodeConfig": {
                                "oauthScopes": [
                                    "https://www.googleapis.com/auth/compute",
                                    "https://www.googleapis.com/auth/devstorage.read_only",
                                    "https://www.googleapis.com/auth/service.management",
                                    "https://www.googleapis.com/auth/servicecontrol",
                                    "https://www.googleapis.com/auth/logging.write",
                                    "https://www.googleapis.com/auth/monitoring",
                                    "https://www.googleapis.com/auth/trace.append",
                                    "https://www.googleapis.com/auth/cloud-platform"
                                ],
                                "machineType": "n1-standard-1",
                                "labels": {
                                    "pool": "api-zero"
                                }
                            },
                            "management": {
                                "autoUpgrade": false,
                                "autoRepair": true
                            },
                            "autoscaling": {
                                "minNodeCount": 1,
                                "maxNodeCount": 20
                            }
                        },
                        {
                            "name": "outbound",
                            "initialNodeCount": 2,
                            "nodeConfig": {
                                "machineType": "custom-1-1024",
                                "oauthScopes": [
                                    "https://www.googleapis.com/auth/compute",
                                    "https://www.googleapis.com/auth/devstorage.read_only",
                                    "https://www.googleapis.com/auth/service.management",
                                    "https://www.googleapis.com/auth/servicecontrol",
                                    "https://www.googleapis.com/auth/logging.write",
                                    "https://www.googleapis.com/auth/monitoring",
                                    "https://www.googleapis.com/auth/trace.append",
                                    "https://www.googleapis.com/auth/cloud-platform"
                                ],
                                "labels": {
                                    "pool": "outbound"
                                }
                            },
                            "management": {
                                "autoUpgrade": false,
                                "autoRepair": true
                            }
                        }
Mikhail Shilkov
  • 34,128
  • 3
  • 68
  • 107
user3270173
  • 31
  • 1
  • 4
  • Do you have any other Ingress Controller (e.g. nginx-ingress) configured in your GKE cluster ? Did you create "kubernetes.io/ingress.class: gce" annotation for 'my-ingress' resource by your self, or was it created automatically ? – Nepomucen Apr 12 '19 at 08:27
  • its part of the ingress definition . ```apiVersion: extensions/v1beta1 kind: Ingress metadata: name: my-ingress annotations: kubernetes.io/ingress.class: "gce"``` – user3270173 Apr 12 '19 at 16:25
  • no other Ingress Controller. the annotation is part of the ingress definition. tried with/without the same result. – user3270173 Apr 12 '19 at 16:39

3 Answers3

3

The author of this post eventually figured out, that issue persist only when cluster is bootstrapped with pulumi.

Nepomucen
  • 4,449
  • 3
  • 9
  • 24
0

It looks like you are missing a default backend (L7 - HTTTP LoadBalancer) for your default ingress controller. From what I observed it`s not deployed when you have Istio add-on enabled in your GKE cluster (Istio has its own default ingress/egress gateways).

Please verify if it`s up and running in your cluster:

kubectl get pod -n kube-system | grep l7-default-backend
Nepomucen
  • 4,449
  • 3
  • 9
  • 24
  • it is present, however don't see any logs ```rg-dev-europe-west4-1> kubectl get pod -n kube-system | grep l7-default-backend l7-default-backend-7ff48cffd7-rk4mq 1/1 Running 0 1h``` – user3270173 Apr 12 '19 at 16:27
  • Do you observe any errors inside loadbalancer pod ? (kubectl logs l7-default-backend-7ff48cffd7-rk4mq) Please find also below a link to troubleshooting guide for gce-ingress. https://github.com/kubernetes/ingress-gce/blob/master/docs/troubleshooting/troubleshooting.md – Nepomucen Apr 15 '19 at 13:07
  • unfortunately, I don't see any logs, continue digging around, thanks – user3270173 Apr 15 '19 at 17:36
  • I forgot to mention the most important thing, the ingress sample from this GKE tutorial is working fine for me. Please try to create a following Ingress resource, with only single path declaration based on the original manifest file example (for me it took couple of minutes for loadbalancer backend to kick in). https://gist.github.com/nepomucen/8318c8d617a98fd70a19cef37fafd82e Output from: http://xx.xxx.xxx.xxx/got-8-premiere Hello, world! Version: 2.0.0 Hostname: hello-world-deployment-5cfbc486f-295hn – Nepomucen Apr 16 '19 at 07:36
  • the issue seems to be in the way I create the cluster. If I create the cluster with console then ingress works just fine. It fails if build the cluster with pulumi. – user3270173 Apr 16 '19 at 14:27
  • You should still be able to create a cluster from pulumi/terraform/not the console and have it work with the ingress. What was the difference between the pulumi and console cluster? – Talador12 Jan 15 '21 at 20:38
0

As per google docs,

In Shared VPC environments or clusters with Network Policies enabled, you must add the annotation cloud.google.com/neg: '{"ingress": true}' to the Service manifest.

In our apps, this required adding the following to service.yaml files

apiVersion: v1
kind: Service
metadata:
  name: myapp
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
Adam Hughes
  • 14,601
  • 12
  • 83
  • 122