kustomize: how to ignore ksops in ci/cd environment?

Question

In my project, we let developers update a repo containing all of the kubernetes manifests. The repo uses kustomize. I've decided to add a validation / lint step to our CI to catch mistakes early.

To do so, I'm trying to run kustomize build on everything in the repo. Where I'm running into trouble is our use of ksops. In this scenario, it's not important to actually decode the secrets. I don't want to install the appropriate key on the CI server or allow it to be pulled. What I'd really like to do is skip all the ksops stuff. I'm looking for something like this (doesn't seems to exist)

kustomize build --ignore-kind=ksops ./apps/myapp/production

If I don't skip the ksops stuff, I get this:

trouble decrypting file Error getting data key: 0 successful groups required, got 0Error: failure in plugin configured via /tmp/kust-plugin-config-24824323; exit status 1: exit status 1

I noticed that someone else thought this was important too. They made a patched version of ksops that can handle my scenario. I'm hoping to do this with the unpatched stuff. Reason: because the folks that come after me will wonder what this is all about.

Update: For reference, I'm doing this in Docker.

Trying out larsks' solution, here's the code I tried:

Dockerfile

FROM alpine

RUN apk add bash curl git

RUN curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash \
    && mv kustomize /usr/bin/kustomize \
    && kustomize version

ENV XDG_CONFIG_HOME=/root/.config
RUN mkdir -p /root/.config/kustomize/plugin

RUN mkdir -p /root/.config/kustomize/plugin/viaduct.ai/v1/ksops \
    && ln -s /bin/true /root/.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
ENV KUSTOMIZE_PLUGIN_HOME=/root/.config/kustomize/plugin

WORKDIR /code

COPY . /code

RUN ./validate.sh

validate.sh

#! /bin/bash

set -e

for i in `find . -name kustomization* -type f | grep -v \/base`; do
    d=`dirname $i`
    echo "$d"
    kustomize build --enable-alpha-plugins "$d"
done

I could have also asked this a different way. "How to ask Kustomize to ignore a kind of resource?" — 101010, Sep 04 '22 at 00:37
Hmm, apparently, removal (ex: of a resource type) is an eschewed feature: https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/ — 101010, Sep 04 '22 at 02:18

larsks · Accepted Answer · 2022-09-05T14:50:52.100

The solution is to create a dummy filter for processing ksops resources. For example, something like this:

mkdir -p fakeplugins/viaduct.ai/v1/ksops
ln -s /bin/true fakeplugins/viaduct.ai/v1/ksops/ksops
export KUSTOMIZE_PLUGIN_HOME=$PWD/fakeplugins

kustomize build --enable-alpha-plugins

This will cause kustomize to call /bin/true when it encounters ksops-encrypted resources. You won't have secrets in your output, but it will generate all other resources.

(The above has been tested with kustomize 4.5.5)

The reason your code is failing is because you're using a Busybox-based Docker image. Busybox is a multi-call binary; it figures out what applet to run based on the name with which it was called. So while on a normal system, we can run ln -s /bin/true /path/to/ksops and then run /path/to/ksops, this won't work in a Busybox environment: it sees that it's being called as ksops and doesn't know what to do.

Fortunately, that's an easy problem to solve:

FROM alpine

RUN apk add bash curl git

RUN curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash \
    && mv kustomize /usr/bin/kustomize \
    && kustomize version

RUN mkdir -p /root/fakeplugins/viaduct.ai/v1/ksops \
    && printf "#!/bin/sh\nexit 0\n" > /root/fakeplugins/viaduct.ai/v1/ksops/ksops \
    && chmod 755 /root/fakeplugins/viaduct.ai/v1/ksops/ksops

ENV KUSTOMIZE_PLUGIN_HOME=/root/fakeplugins

COPY validate.sh /bin/validate-overlays

WORKDIR /code

And now, given a layout like this:

.
├── Dockerfile
├── example
│   ├── base
│   │   ├── deployment.yaml
│   │   ├── kustomization.yaml
│   │   └── pvc.yaml
│   └── overlay
│       ├── deployment_patch.yaml
│       ├── kustomization.yaml
│       ├── pg-password.enc.yaml
│       └── secret-generator.yaml
└── validate.sh

I can run from the top directory:

docker run --rm -v $PWD:/code my-kustomize-image validate-overlays

NB: I've slightly modified validate.sh to do the filtering in find rather than piping the output to grep -v :

#!/bin/bash

set -e

find . -name base -prune -o -name kustomization.yaml -print |
    while read -r overlay; do
        overlay="${overlay%/*}"
        echo "$overlay"
        kustomize build --enable-alpha-plugins "$overlay"
    done

I'm getting `ksops: applet not found`. I'm doing this in docker, if that matters. I've experimented with your code -- no change. I'm on Kustomize v4.5.7 — 101010, Sep 04 '22 at 22:16
Which docker image are you using? And how exactly are you running kustomize? In your question you show `kustomize build --ignore-kind=ksops ./apps/myapp/production`, but that doesn't involve `docker run` at all. — larsks, Sep 04 '22 at 22:26
I've updated the answer with a functional Dockerfile that implements this solution. — larsks, Sep 05 '22 at 14:51

kustomize: how to ignore ksops in ci/cd environment?

1 Answers1