1

Starting kubernetes 1.20 there has been a change regarding ownerReferences and how K8s performs GC. Basically if a resource in x namespace spins up a pod/job in a y namespace with child having ownerReferences referencing to parent resource in x, K8s terminates the child pod/job.

Reference:

  • Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.
    • A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
    • A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]

If we remove the ownerReferences, the resource wont be garbage collected. Is there a way to deal with this situation i.e.; how to make ownerReferences work in multiple namespaces OR let the job/pod clean itself once completed? Thanks.

Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
colossal
  • 529
  • 1
  • 5
  • 22
  • Could you describe your use case in more details? How you've got a child object in another namespace from his parent? Why you need it to work in that way? – VAS Jun 11 '21 at 14:41
  • Consider a resource creating another resource. While defining a resource you can specify which namespace the resource will go in. Its just how the cluster is setup that generated the need to create resources in different namespace based on the project. – colossal Jun 11 '21 at 17:30
  • So why do you use an OwnerReference? You can use labels – VAS Jun 11 '21 at 17:59
  • I used `ownerReferences` so that when parent object is deleted, the child is also garbage collected. Otherwise pods will be hanging around. Can this be achieved by labels ? – colossal Jun 11 '21 at 18:49
  • A real world example of using labels for specifying an object owner: https://github.com/istio/istio/issues/29430#issuecomment-739636850 – VAS Jun 21 '21 at 08:11

1 Answers1

1

As per Fix GC uid races and handling of conflicting ownerReferences #92743

namespaces are intended to be independent of each other, so cross-namespace references have not been permitted in things like ownerReferences, secret/configmap volume references, etc.

additionally, granting permissions to namespace a is not generally intended to provide visibility or ability to interact with objects from namespace b (or cause system controllers to interact with objects from namespace b).

and Update GC cross-namespace note #25091

Cross-namespace owner references are disallowed by design.

So, using ownerReferences for garbage collection across namespaces is not possible by desing.

However, you can emulate multi-namespace GC using labels. You just need to configure those labels when some object creates sub-object.

Alternatively you can delete a namespace to GC all object in that namespace, but that's probably suboptimal solution.

EDIT

$ kubectl label pods owner=my -l region=europe
$ kubectl label pods owner=my -l region=pacific

$ kubectl label svc owner=my -l svc=europe
$ kubectl label svc owner=my -l svc=pacific

$ kubectl label pod kube-proxy-2wpz2 owner=my -n kube-system 
$ kubectl label pod kube-proxy-cpqxt  owner=my -n kube-system 

$ kubectl get pods,svc -l owner=my --show-labels --all-namespaces

NAMESPACE     NAME                   READY   STATUS        RESTARTS   AGE    LABELS
default       pod/aloha-pod          1/1     Running       0          54d    app=aloha,owner=my,region=pacific
default       pod/ciao-pod           1/1     Running       0          54d    app=ciao,owner=my,region=europe
default       pod/hello-pod          1/1     Terminating   0          54d    app=hello,owner=my,region=europe
default       pod/ohayo-pod          1/1     Running       0          54d    app=ohayo,owner=my,region=pacific
kube-system   pod/kube-proxy-2wpz2   1/1     Running       2          299d   controller-revision-hash=5cf956ffcf,k8s-app=kube-proxy,owner=my,pod-template-generation=1
kube-system   pod/kube-proxy-cpqxt   1/1     Running       3          299d   controller-revision-hash=5cf956ffcf,k8s-app=kube-proxy,owner=my,pod-template-generation=1

NAMESPACE   NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   LABELS
default     service/europe    ClusterIP   10.109.5.102    <none>        80/TCP    54d   owner=my,svc=europe
default     service/pacific   ClusterIP   10.99.255.196   <none>        80/TCP    54d   owner=my,svc=pacific

$ kubectl delete pod,svc -l owner=my --dry-run --all-namespaces

pod "aloha-pod" deleted (dry run)
pod "ciao-pod" deleted (dry run)
pod "hello-pod" deleted (dry run)
pod "ohayo-pod" deleted (dry run)
pod "kube-proxy-2wpz2" deleted (dry run)
pod "kube-proxy-cpqxt" deleted (dry run)
service "europe" deleted (dry run)
service "pacific" deleted (dry run)

Alternatively there could be a bash script that deletes all objects whose controller object doesn't exist, based on labels. It could also run inside the cluster with proper service account configured.

There is no straightforward, built-in option to achieve what you want. You should keep owner referenced objects in the same namespace.