GitOps: Roll out changes to k8s resources across multiple namespaces

Question

We have a multi-tenant kubernetes cluster, that hosts various customer environments. Kubernetes namespaces are used to isolate these customer environments. Each of these namespaces have a similar set of k8s resources (deployments, configmap etc) configured. Some of these resources are identical across namespaces, while some are not identical.

From a continuous delivery perspective, I am exploring options to roll out changes to these identical components (across namespaces) in a seamless manner. Git-Ops/Pull-based-Continuous-Delivery seems to be a good approach, which would enable us to seamlessly manage 100s of namespaces across various clusters.

Explored a couple of Git-Ops tools such as ArgoCD, Fluxv2 etc... but couldn't figure out whether these tools would let you roll out changes to multiple namespaces simultaneously (or within a predictable time window). It will be helpful, if you can guide/advice me in terms of choosing the right tool/approach to perform roll out to multiple namespaces. It will also be good to understand whether these Git-Ops tools can be customised to handle such scenarios.

An illustration of how namespaces are setup in our k8s cluster.

Namespace: customer1
Deployments: app1, common-app, common-service
Configmaps: cm1, common-cm

Namespace: customer2
Deployments: app2, common-app, common-service
Configmaps: cm2, common-cm

common-app, common-service, common-cm are identical across environments/namespaces.

app1, cm1, app2, cm2 are not identical. They have different image tags, labels etc.

Camil · Accepted Answer · 2021-06-18T12:06:14.360

1

I'm using Fluxv2. My approach will be to create charts for common-app, common-service, etc with same values and define the namespace on separate HelmReleases

So you can put the common-app chart on clusters/production/charts/common-apps and refer the chart path on multiple HelmReleases which can look like this

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: common-apps-customer1
  namespace: customer1
  labels:
    chart: common-apps
spec:
  interval: 1m
  releaseName: common-apps
  chart:
    spec:
      chart: clusters/production/charts/common-apps
      sourceRef:
        kind: GitRepository
        name: manifests
        namespace: flux-system
  values:
    some_key: some_value
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: common-apps-customer2
  namespace: customer2
  labels:
    chart: common-apps
spec:
  interval: 1m
  releaseName: common-apps
  chart:
    spec:
      chart: clusters/production/charts/common-apps
      sourceRef:
        kind: GitRepository
        name: manifests
        namespace: flux-system
  values:
    some_key: some_value

Update

for ns in $(kubectl get ns --no-headers | cut -d " " -f1); do    
    flux create helmrelease common-apps-$ns \
    --source=GitRepository/manifests \
    --chart=common-apps \
    --release-name=common-apps \
    --target-namespace=$ns \
    --interval=1m0s \
    --export > ./clusters/production/common-apps-$ns.yaml     
done;

This will generate the necessary manifests that should be anyway synced with your repo.

edited Jun 18 '21 at 12:06

answered Jun 18 '21 at 01:07

Camil

7,800
2
25
28

So, the approach would be to create a `HelmRelease` per namespace and point them to the same git repo. This would ensure that all namespaces end up applying the changes from the same git repo (my understanding). This would work well for me. But I am wondering whether we could do it in a way, which would avoid creation of multiple namespace specific `HelmRelease` resources (I have 100s of namespaces in the cluster; would prefer to avoid duplication). Is there a way to customise the flow, to read the namespaces from the cluster (on the fly) & apply the changes? – dinup24 Jun 18 '21 at 08:21
Not directly in Flux. I believe you're looking for something similar to this: https://github.com/fluxcd/flux2/discussions/872 – Camil Jun 18 '21 at 11:59
Bu I would do it like this: – Camil Jun 18 '21 at 12:00
`for ns in $(kubectl get ns --no-headers | cut -d " " -f1); do flux create helmrelease common-apps-$ns \ --source=GitRepository/manifests \ --chart=common-apps \ --release-name=common-apps \ --target-namespace=$ns \ --interval=1m0s \ --export > ./clusters/production/common-apps-$ns.yaml done;` – Camil Jun 18 '21 at 12:00
that will generate the necessary config. i will edit the original post for proper formatting – Camil Jun 18 '21 at 12:02
ah nice. It helps... I will accept your answer, as it solves my problem. – dinup24 Jun 21 '21 at 08:41

score 0 · Answer 2 · answered Jun 17 '21 at 22:18

We use Flux to deploy resources to our K8s cluster.

these tools would let you roll out changes to multiple namespaces simultaneously (or within a predictable time window)

Yeah, Flux basically monitors your git repo for any changes and applies them to the cluster. By default Flux checks for changes every 5 minutes but that can be reduced (though I remember reading that anything lower than a minute can be problematic).

If the changes are made to more than one namespace it will apply all of them in one go. I haven't ever had to see if all changes to multiple namespaces are applied simultaneously as they are generally pretty quick. However I must say, we haven't tested it across "100s of namespaces" as you write but it works just fine across a handful of them.

GitOps: Roll out changes to k8s resources across multiple namespaces

2 Answers2