Enforce a domain pattern that a service can use

Question

I have a multi-tenant Kubernetes cluster. On it I have an nginx reverse proxy with load balancer and the domain *.example.com points to its IP.

Now, several namespaces are essentially grouped together as project A and project B (according to the different users).

How, can I ensure that any service in a namespace with label project=a, can have any domain like my-service.project-a.example.com, but not something like my-service.project-b.example.com or my-service.example.com? Please keep in mind, that I use NetworkPolicies to isolate the communication between the different projects, though communication with the nginx namespace and the reverse proxy is always possible.

Any ideas would be very welcome.

---

EDIT:

I made some progress as have been deploying Gatekeeper to my GKE clusters via Helm charts. Then I was trying to ensure that only Ingress hosts of the form ".project-name.example.com" should be allowed. For this, I have different namespaces that each have labels "project=a" or similar and each of these should only allow to use ingress of the form ".a.example.com". Hence I need that project label information for the respective namespaces. I wanted to deploy the following resources

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredingress
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredIngress
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredingress

        operations := {"CREATE", "UPDATE"}
        ns := input.review.object.metadata.namespace


        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          input.request.kind.kind == "Ingress"
          not data.kubernetes.namespaces[ns].labels.project

          msg := sprintf("Ingress denied as namespace '%v' is missing 'project' label", [ns])
        }


        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          input.request.kind.kind == "Ingress"
          operations[input.request.operation]

          host := input.request.object.spec.rules[_].host
          project := data.kubernetes.namespaces[ns].labels.project
          not fqdn_matches(host, project)
          msg := sprintf("invalid ingress host %v, has to be of the form *.%v.example.com", [host, project])

        }

        fqdn_matches(str, pattern) {
          str_parts := split(str, ".")
          count(str_parts) == 4
          str_parts[1] == pattern
        }

---

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredIngress
metadata:
  name: ns-must-have-gk
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Ingress"]

---

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: "gatekeeper-system"
spec:
  sync:
    syncOnly:
      - group: ""
        version: "v1"
        kind: "Namespace"

However, when I try to setup everything in the cluster I keep getting:

kubectl apply -f constraint_template.yaml

Error from server: error when creating "constraint_template.yaml": admission webhook "validation.gatekeeper.sh" denied the request: invalid ConstraintTemplate: invalid data references: check refs failed on module {template}: errors (2):
disallowed ref data.kubernetes.namespaces[ns].labels.project
disallowed ref data.kubernetes.namespaces[ns].labels.project

Do you know how to fix that and what I did wrong. Also, in case you happen to know a better approach just let me know.

Answer 1

Alternative to other answer, you may use validation webhook to enfore by any parameter present in the request. Example, name,namespace, annotations, spec etc.

The validation webhook could be a service running in the cluster or External to cluster. This service would essentially make a logical decision based on the logic we put. For every request Sent by user, api server send a review request to the webhook and the validation webhook would either approve or reject the review.

You can read more about it here, more descriptive post by me here.

Answer 2

If you want to enforce this rule on k8s object such as configmap or ingress, I think you can use something like OPA

In Kubernetes, Admission Controllers enforce semantic validation of objects during create, update, and delete operations. With OPA you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.

reference