How to create a kustomize common sidecar component

Question

On a Kubernetes cluster, I have multiple Deployment resources. For security, I am using a sidecar proxy pattern where the Service will proxy traffic to the sidecar, which will ensure authentication before passing on to the deployed application.

I am trying to set up Kustomize to do this. Since the sidecar definition is likely environment specific, I don't want to include the sidecar in my base manifests, but would like it to be an overlay. Since I have multiple deployments that will need to attach that sidecar, it seemed appropriate to have the sidecar specification be a common shared component. This seemed like appropriate use of the Kustomize Component resource, but perhaps I'm wrong.

I have something similar to the following:

.
├── base
│   ├── app1
│   │   ├── deployment.yaml
│   │   └── kustomization.yaml
│   ├── app2
│   │   ├── deployment.yaml
│   │   └── kustomization.yaml
│   └── app3
│       ├── deployment.yaml
│       └── kustomization.yaml
├── components
│   └── sidecar
│       ├── deployment-sidecar.yaml
│       └── kustomization.yaml
└── overlays
    └── dev
        └── kustomization.yaml

I'd like the sidecar component to be applied to the 3 app deployments, but I can't seem to find a way to do this. Am I misusing components here?

My components/sidecar/kustomization.yaml file looks like:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
  - path: deployment-sidecar.yaml
    target:
      labelSelector: xxx

This works, however it specifies the target of the patch in the component, whereas I would like to leave the component more generic and instead specify the target in overlays/dev.

Is there a better way to be handling this? In summary, I want the overlay to be able to define when the sidecar should be added, and to which specific deployments to add it to.

larsks · Accepted Answer · 2023-02-23T20:35:06.157

In summary, I want the overlay to be able to define when the sidecar should be added, and to which specific deployments to add it to.

My first thought was that you could have a label that means "apply the sidecar patch", and use that in the Component:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
  - path: deployment-sidecar.yaml
    target:
      labelSelector: "inject-sidecar=true"

And then in your overlay(s), use a patch to apply that label to specific deployments:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base

components:
  - ../../sidecar

patches:
  - target:
      kind: Deployment
      labelSelector: "app=app1"
    patch: |
      - op: add
        path: /metadata/labels/inject-sidecar
        value: "true"

Unfortunately, this won't work because patches are applied after processing all resources and components.

We can still do this, but it requires an intermediate stage. We can get that by creating another component inside the dev overlay that is responsible for applying the labels. In overlays/dev/apply-labels/kustomization.yaml you have a kustomization.yaml that contains the logic for applying the inject-sidecar label to specific Deployments (using a label selector, name patterns, or other criteria):

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
  - target:
      kind: Deployment
      labelSelector: "app=app1"
    patch: |
      - op: add
        path: /metadata/labels/inject-sidecar
        value: "true"

And then in overlays/dev/kustomization.yaml you have:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

components:
  - apply-labels
  - ../../components/sidecar

This gets you what you want:

The sidecar patch is specified in a single place
Your overlay determines to which deployments you apply the sidecar patch

There's a level of complexity here that is only necessary if:

You have multiple overlays
You want to selectively apply the sidecar only to some deployments
You want the overlay to control to which deployments the patch is applied

If any of those things aren't true you can simplify the configuration.

Thanks for the detailed answer. I attempted exactly what you initially suggested and discovered that it didn't work because of the order of execution, as you noted. So, thank you for the way to get around that with the intermediate stage. It is a bit complex, but I suppose there isn't a better approach. — Mike, Feb 23 '23 at 20:53

How to create a kustomize common sidecar component

1 Answers1