0

I've a namespace I'm unable to delete in my Kubernetes cluster. When I run kubectl get ns traefik -o yaml, I get the following:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2021-06-11T20:28:59Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2021-06-11T20:29:00Z"}]}'
    field.cattle.io/projectId: c-5g2hz:p-bl9sf
    lifecycle.cattle.io/create.namespace-auth: "true"
  creationTimestamp: "2021-06-11T20:28:58Z"
  deletionTimestamp: "2021-07-04T07:21:20Z"
  labels:
    field.cattle.io/projectId: p-bl9sf
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:field.cattle.io/projectId: {}
        f:labels:
          .: {}
          f:field.cattle.io/projectId: {}
      f:status:
        f:phase: {}
    manager: agent
    operation: Update
    time: "2021-06-11T20:28:58Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cattle.io/status: {}
          f:lifecycle.cattle.io/create.namespace-auth: {}
    manager: rancher
    operation: Update
    time: "2021-06-11T20:28:58Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          .: {}
          k:{"type":"NamespaceContentRemaining"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"NamespaceDeletionContentFailure"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"NamespaceDeletionDiscoveryFailure"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"NamespaceDeletionGroupVersionParsingFailure"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"NamespaceFinalizersRemaining"}:
            .: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-07-04T07:21:26Z"
  name: traefik
  resourceVersion: "15400692"
  uid: 4b198956-bbd5-4bdb-9dc6-9d53feda91e4
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2021-07-04T07:21:25Z"
    message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
      complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently
      unable to handle the request'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2021-07-04T07:21:26Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2021-07-04T07:21:26Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2021-07-04T07:21:26Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2021-07-04T07:21:26Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

And when I run kubectl delete ns traefik --v=10, the last output is the following:

I0708 18:38:26.538676   31537 round_trippers.go:425] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" 'http://127.0.0.1:44427/6614317c-41da-462b-8be3-c6cda2f0df24/api/v1/namespaces?fieldSelector=metadata.name%3Dtraefik&resourceVersion=17101173&watch=true'
I0708 18:38:27.013394   31537 round_trippers.go:445] GET http://127.0.0.1:44427/6614317c-41da-462b-8be3-c6cda2f0df24/api/v1/namespaces?fieldSelector=metadata.name%3Dtraefik&resourceVersion=17101173&watch=true 200 OK in 474 milliseconds
I0708 18:38:27.013421   31537 round_trippers.go:451] Response Headers:
I0708 18:38:27.013427   31537 round_trippers.go:454]     Access-Control-Allow-Origin: *
I0708 18:38:27.013450   31537 round_trippers.go:454]     Date: Thu, 08 Jul 2021 16:38:27 GMT
I0708 18:38:27.013453   31537 round_trippers.go:454]     Connection: keep-alive
I0708 18:38:27.013468   31537 request.go:708] Unexpected content type from the server: "": mime: no media type

I already tried to remove the finalizers as described on https://www.ibm.com/docs/en/cloud-private/3.2.0?topic=console-namespace-is-stuck-in-terminating-state but after some seconds I finally get EOF:

> curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/traefik/finalize
EOF

Is anyone having any idea how I can delete the traefik namespace?

Nrgyzer
  • 783
  • 1
  • 14
  • 38
  • Can you check, if something else is inside the namespace? `for RES in $(kubectl api-resources --namespaced=true -o name); do echo $RES; kubectl get $RES -o name; done` – Manuel Jul 08 '21 at 21:28
  • I guess it's about `Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request`. Please check [this answer](https://stackoverflow.com/a/62464015/15537201) – moonkotte Jul 09 '21 at 07:44
  • @Manuel It shouldn't be the case, because you can delete namespace even if there are resources inside - they will be gone with the namespace - [source](https://kubernetes.io/docs/tasks/administer-cluster/namespaces/#deleting-a-namespace) – moonkotte Jul 09 '21 at 07:46
  • Also `after some seconds I finally get EOF:` - check your json in validator just in case. – moonkotte Jul 09 '21 at 07:52
  • 1
    I know, that deleting the namespace should delete all other resources inside. I'm just mentioning it, because i had once an issue, that another resource was not able to get deleted. So the namespace deletion stucked as well. – Manuel Jul 09 '21 at 08:21
  • 1
    Thanks to everyone for all these ideas. I fixed the issue for my Metrics server (the problem you posted @moonkotte) and finally, the namespace were delete. – Nrgyzer Jul 09 '21 at 14:50

1 Answers1

2

Posting this as a community wiki out of comments, feel free to edit and expand.

After analysing a state of the problematic namespace, this part was the main cause of the issue:

message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
  complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently
  unable to handle the request'

The issue was with metric server in kubernetes. Once the metric server is available, namespace can be unblocked to be deleted.

Similar issue was resolved in another answer on the stackoverflow.

moonkotte
  • 3,661
  • 2
  • 10
  • 25