How do I enable / disable the multi tenancy restrictions in Flux2 with AKS (and should I?)

Question

I'm starting on a journey with Flux2 against my shiny new AKS cluster (testing/development) and wanted to use GitOps as my default way of deploying stuff into my cluster. As a result, i thought I'd use Bicep to create both the cluster and set up my initial Gitops with flux. So far so good.

I arbitrarily decided to use 'flux-gitops' as my namespace for any GitRepository and Kustomization mappings forward going. As such wanted to do just one mapping and then apply all other changes within the scope of Flux itself.

I set this up using bicep like this:

resource fluxConfiguration 'Microsoft.KubernetesConfiguration/fluxConfigurations@2023-05-01' = {
name: 'flux-configuration'
scope: aksCluster
properties: {
scope: 'cluster'
namespace: 'flux-gitops'
sourceKind: 'GitRepository'
suspend: false

gitRepository: {
  url: 'https://myrepo/_git/flux.orchestration'
  timeoutInSeconds: 600
  syncIntervalInSeconds: 600
  localAuthRef: 'flux-configuration-protected-parameters'
  repositoryRef: {
    branch: 'main'
  }
}

And then switched to the git repo itself. My goal was not lofty, it was to install Nginx Ingress Controller via helm. Firstly finding the helm chart repository wasn't straightforward so it's entirely possible that itself is wrong. But I set up a HelmRepository type for flux as follows:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: ingress-nginx
  namespace: flux-system
spec:
  interval: 30m
  url: https://kubernetes.github.io/ingress-nginx

Again be mindful that the chart repo might be out of date? I've tried a few variants. Also, what are the namespaces in this particular case? Is this where the reference for the flux system knows where this reference to a helm repository lives? It's confusing to me... Then:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  interval: 5m
  chart:
    spec:
      chart: ingress-nginx
      version: '4.0.13'
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
        namespace: flux-system
      interval: 1m

The release.

There are namespaces and names all over the shop here. And Flux examples themselves use the same word for everything making it hard to differentiate what is what. In the above release, the source ref has a HelmRepository called 'ingress-nginx' that I've put in the flux-system right? Presumably if I've put the repository ref in another namespace I would adjust this namespace accordingly?

As for the release itself, the name and namespace are equally confusing at the top level. What are these? Is this the name of the release? The name of the chart (we see that further down) and the namespace is that the destination for the release? I don't know. But the above example does not work for me.

I get the cross namespace references are not supported. So the crux of the question is, how do I enable/disable that in Bicep (and should I) and equally, what are all these different names and namespaces and how do I make sense of them? I've tried a lot of variations and this is the closest I've got to it working.

score 0 · Answer 1 · answered Aug 07 '23 at 10:22

In answer to my own question, some things I've learnt the hard way over the weekend.

Don't try and disable the multi-tenancy features that have been introduced to the FLUX system, they are intended to keep your environment secure.
The multi-tenancy / cross-namespace restriction seems to be relatively new meaning that a lot of previous blog posts will actually fail to work even if you copy and paste them verbatim so look out for that.
Keep your HelmRepository and HelmRelease definitions in the same namespace as your flux definitions. I've created a namespace called flux-gitops for example.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: ingress-nginx
  namespace: flux-gitops
spec:
  interval: 24h
  url: https://kubernetes.github.io/ingress-nginx

with a matching release

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: flux-gitops
spec:
  interval: 30m
  targetNamespace: ingress-nginx
  chart:
    spec:
      chart: ingress-nginx
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
        namespace: flux-gitops
      interval: 12h
  values:
    controller:
      service:
        type: LoadBalancer
    admissionWebhooks:
      enabled: false

Most importantly, define a target namespace. This is the bit from the above healm release that will save you from the craziness:

targetNamespace: ingress-nginx

Which then allows for the helm release to complete without that really unclear message saying the chart is not ready.

How do I enable / disable the multi tenancy restrictions in Flux2 with AKS (and should I?)

1 Answers1