I have a use case that calls for a large number of very similar Deployments. Each will differ only in name/labels and a couple of environment variables. New ones will be frequently created, and existing ones will occasionally be deleted. Very occasionally, a change will be made to the template from which they're all derived - e.g. an update to a container image version - and all existing Deployments should be modified when that happens. I'm looking for a solution to automate this.
It's also worth highlighting that I'll have multiple clusters. Each will have its own, non-overlapping set of these Deployments, but all derived from the same template.
The best solution I've come up with so far is to use a combination of Flux and its Kustomize controller. A git repo contains a base Kustomization with the template Deployment, and a directory per cluster. A cluster's directory then contains a subdirectory per Deployment, with a kustomization.yaml that pulls in the base and patches it appropriately, and kustomize create --autodetect --recursive run at the cluster directory level then creates a kustomization.yaml that includes all of the Deployments for that cluster.
Within each cluster, Flux is installed, and a combo of a GitRepository and Kustomization objects to grab the per-cluster kustomization for that cluster. That will create all of this cluster's Deployments.
New Deployments are then created by creating a new subdirectory & kustomization.yaml in the Git repo, and regenerating the cluster-scope kustomization.yaml (the latter would probably be done by CI pipeline). Existing ones are deleted by deleting the corresponding subdirectory. And universal template changes are applied just by changing the base Kustomization in the repo.
This also allows for one cluster to be a "canary" for testing changes to the base template, by having that cluster point at a different branch of the repo.
I think this ought to work, so my questions are (a) can anyone see any problems with it, and (b) can anyone suggest anything better?
