Kubebuilder: resource deletion not having expected side effects

Question

I'm creating a custom resource definition (CRD) with an associated controller using kubebuilder. My controller reconcile loop creates a deployment sub-resource and parents it to the custom resource using controllerutil.SetControllerReference(&myResource, deployment, r.Scheme). I've also configured my reconciler so "own" the sub-resource, as follows:

// SetupWithManager sets up the controller with the Manager.
func (r *MyResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&mygroupv1alpha1.MyResource{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

However, when I run my controller locally using make run, I noticed that deleting the my CR (the root object) doesn't cause the Deployment sub-resource to get garbage collected. I also noticed that deleting the Deployment sub-resource doesn't trigger my reconciler to run. Why is this? Is there something I'm not doing or is this possibly a limitation of local development/testing?

Check the `metadata.ownerReferences` on the deployment, do you see the ref? If not, you might not be setting it on the right object or doing it too late to affect the object when it is created. — coderanger, Aug 01 '21 at 02:57
@coderanger thanks for the hint! I didn't see any `ownerReferences` field under my deployment's `metadata` section. I double-checked my code and sure enough, I was setting it too late (after I created it)! After fixing the code, things seem to be working! I'll post an answer with the details. — Chris Gillum, Aug 02 '21 at 16:25

score 3 · Accepted Answer · answered Aug 02 '21 at 16:42

Using @coderanger's hint, I could see that the metadata.ownerReferences weren't being set correctly when running the following command:

kubectl get deployments sample-deployment -o yaml

The problem was my controller's reconcile code. I was calling controllerutil.SetControllerReference(&myResource, deployment, r.Scheme) only after I'd already created and persisted the Deployment.

Buggy code

log.Info("Creating a deployment")

deployment := &appsv1.Deployment{
    ObjectMeta: metav1.ObjectMeta{
        Name:      deploymentName,
        Namespace: myResource.Namespace,
    },
    Spec: deploymentSpec,
}

if err = r.Create(ctx, deployment); err != nil {
    log.Error(err, "Failed to create deployment")
    if errors.IsInvalid(err) {
        // Don't retry on validation errors
        err = nil
    }
    return ctrl.Result{}, err
}

// Establish the parent-child relationship between my resource and the deployment
log.Info("Making my resource a parent of the deployment")
if err = controllerutil.SetControllerReference(&myResource, deployment, r.Scheme); err != nil {
    log.Error(err, "Failed to set deployment controller reference")
    return ctrl.Result{}, err
}

To fix it, I needed to swap the order of the call to r.Create and controllerutil.SetControllerReference:

Working code

log.Info("Creating a deployment")

deployment := &appsv1.Deployment{
    ObjectMeta: metav1.ObjectMeta{
        Name:      deploymentName,
        Namespace: myResource.Namespace,
    },
    Spec: deploymentSpec,
}

// Establish the parent-child relationship between my resource and the deployment
log.Info("Making my resource a parent of the deployment")
if err = controllerutil.SetControllerReference(&myResource, deployment, r.Scheme); err != nil {
    log.Error(err, "Failed to set deployment controller reference")
    return ctrl.Result{}, err
}

// Create the deployment with the parent/child relationship configured
if err = r.Create(ctx, deployment); err != nil {
    log.Error(err, "Failed to create deployment")
    if errors.IsInvalid(err) {
        // Don't retry on validation errors
        err = nil
    }
    return ctrl.Result{}, err
}

I was able to confirm that this worked by looking at the metadata.ownerReferences YAML data for my created deployment (using the command referenced above).

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2021-08-02T16:22:04Z"
  generation: 1
  name: sample-deployment
  namespace: default
  ownerReferences:
  - apiVersion: resources.mydomain.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: MyResource
    name: myresource-sample
    uid: 6ebb146c-afc7-4601-bd75-58efc29beac9
  resourceVersion: "569913"
  uid: d9a4496f-7418-4831-ab87-4804dcd1f8aa

Kubebuilder: resource deletion not having expected side effects

1 Answers1