Is there a DRY way to patch multiple resources using kubectl v1.17?

Question

I'm working under some restrictions, outside of my control, namely the CI/CD pipeline and the infrastructure in general.

I've been testing my kustomizations on my local console which has v1.21, but my pipeline is running v1.17.

So components are out, and apparently so is patching multiple resources with the patches object: https://github.com/kubernetes-sigs/kustomize/issues/1373#issuecomment-618439078 And no, directly invoking kustomize (of any version) is not available either.

Not using components, I can deal with, but multi-patching support seems incredibly basic.

Here's a basic example of what I was doing:

    root
    |- kube
       |- kustomize
            |- base
            |    |- job_dir
            |    |   |- job1.yml
            |    |   |- ..
            |    |- job.patch.yml 
            |    |   # config that applies to some jobs
            |    |- kustomization.yml
            |         # resources:
            |         #   - job_dir/job1.yml
            |         #   - ..
            |         #     
            |         # patches:
            |         #   - target:
            |         #       kind: Job
            |         #     patch: |-
            |         #       - op: add
            |         #         ..
            |         #   - path: job.patch.yml
            |         #     target:
            |         #       kind: Job
            |         #       labelSelector: patchWith=job.patch
            |- overlays
                |- dev
                |   |- kustomization.yml
                |        # images: ..
                |- prod
                    |- kustomization.yml
                         # images: ..

executed with kubectl 1.17:

$ kubectl apply -k kube/kustomize/base
error: json: cannot unmarshal object into Go struct field Kustomization.patchesStrategicMerge of type patch.StrategicMerge

executed with kubectl 1.20+:

kubectl apply -k kube/kustomize/base
job.batch/job1 created
job.batch/..

Without multi-patching, can I emulate the same behavior somehow?

Preferably with using some templating tool like jinja2.

Worth noting that the infrastructure's actual Kubernetes instance is running v1.20, so there's some restrictions there, too. (for instance, why I'm not just using indexed jobs ಠ_ಠ) So, if there's a way to

Answer 1

I found a way. This probably isn't the best way.

You can use kubectl patch to patch specific files -f PATH, directories -f DIR_PATH, kustomizations -k DIR_PATH, or directories recursively -Rf DIR_PATH.

You should be careful, though, as this applies the patch to everything in scope.

For instance, this is what happens when a ConfigMap is in the directory where you're applying a patch meant for jobs:

$ ls kube/kustomize/base
job.yml config.yml
$ patch='[{op: add, path: /spec/template/spec/restartPolicy, value: Never}]'
$ kubectl patch --local=true --type=json -p "$patch" -o name -R kube/kustomize/base
job.batch/job1
error: add operation does not apply: doc is missing path: "/spec/template/spec/restartPolicy": missing value

In this case, it's not too bad since nothing unexpected got the patch, but if you're not careful you might apply a patch to the wrong resources. Globs don't work, so you won't be able to patch based on a file pattern unless there's only one resource that matches.

So, -f PATH is the best, and safest.

You can compile a list of the correct files using something like this, depending on the system:

$ kust_dir="kube/kustomize/base"
$ kust_file="${kust_dir}/kustomization.yml"
$ sed -n '/^resources:\s*$/,/^[^ \t-]/p' ${kust_file} | tail -n +2 | head -n -1 | sed -i "s@\s*-\s\+@$kust_dir@" > manifest_list.tmp

It gets pretty complicated if you want a bulletproof way to use label selectors, but this flawed approach works for most things:

# recursive look for label anywhere in $kust_dir, output filenames, filter output by the existing list of filenames
$ grep -lre "$label:\s*$value" $kust_dir | grep -Ff manifest_list.tmp > filtered_by_label.tmp

As stated, it matches any label in the file. If you know the whitespace string that manifests are indented with, and they're uniform, can do this:

# indent_str="  " if it's two spaces
$ grep -lre "\($indent_str\)\{2\}$label:\s*$value" $kust_dir | grep -Ff manifest_list.tmp > filtered_by_label.tmp

which should only match the first label. Use -z and -P options to match multiple lines to get a better result, but it may not work on every system, and you still might need to know the number of tabs, or else use some find -exec awk or find -exec sed.

---

once you have a file list you're confident of, do something like this:

$ for file in $file_list; do
>   kubectl patch --local=true --type=json -p '$patch_str' -o yaml -f $file > $file
> done
$ kubectl apply -k kube/kustomize/base
job/job1 created

You can use -R -f kube/kustomize/base/jobs or -k kube/kustomize/base to apply the patch to multiple resources, but the output won't include the --- separator, so you can't write the output to a file and then apply it without some processing. But you can tinker with that as a possible alternative.