Kubernetes kustomize with prometheus

Question

I have multiple environments where we define prometheus alerts, and these alerts have the same labels and description among environments, they just differ in the expression and priority.

I am currently using Kustomize with the following structure:

base/prometheus/ It contains 2 files, kustomization and servicemonitor.
overlays/environment/ It contains a kustomization and the current alerting rules

This is the kustomation file in the overlay folder:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

...

bases:
  ...
  - ../../base/prometheus/

resources:
  - alerting-rules.yaml

patchesStrategicMerge:
  ...

And the one in the base:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - service-monitor.yaml

I tried to add the rules in the base:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - service-monitor.yaml
  - prometheus-rules.yaml

And then add it in the overlays, adding this new resource to the patchesStrategicMerge, but when I run kubectl kustomize, the resulting rule is always the one that I put in the overlay, without the information (labels and description) from the base.

Edit: These are my files:

base/prometheus-alerts/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-workspace

resources:
  - alerting-rules-test.yaml

base/prometheus-alerts/alerting-rules-test.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: alerting-rules-test
  namespace: my-namespace
  labels:
    app-id: MY-APP-ID
    ms-id: my-ms-id
spec:
  groups:
    - name: alerting-rules-test
      interval: 60s
      rules:
        - alert: alert-testing
          expr: absent(application:health_microservice{service="my-service"})
          for: 10m
          labels:
            priority: "MAJOR"
            mc_id: ""
            event_id: "100"
            namespace: "my-namespace"
          annotations:
            summary: >
              Summary of test alert
            description: >
              Description of test alert

overlays/env/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-namespace

commonAnnotations: #use by dynatrace
  ... # some labels

bases:
  ... # other bases
  - ../../base/prometheus-alerts

resources:
  ... # some resources from the same folder

patches:
  ...
  - alerting-rules-test.yaml

overlays/env/alerting-rules-test.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: alerting-rules-test
  namespace: my-namespace
spec:
  groups:
    - name: alerting-rules-test
      interval: 60s
      rules:
        - alert: alert-testing
          expr: absent(application:health_microservice{service="my-service"})
          labels:
            priority: "MINOR"

When I run kubectl kustomize, this is the resulting yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    ... #annotations from env/kustomization.yaml
  labels:
    app-id: MY-APP-ID
    ms-id: my-ms-id
  name: alerting-rules-test
  namespace: my-namespace
spec:
  groups:
  - interval: 60s
    name: alerting-rules-test
    rules:
    - alert: alert-testing
      expr: absent(application:health_microservice{service="my-service"})
      labels:
        priority: MINOR

The problem is that the complete rule gets replaced by the one defined en env/alerting-rules-test.yaml. The resulting PrometheusRule should have the other fields defined in the base, like the summary and the description.

I have tried with JsonPatches6902 but it does not replace the value.

In the overlays/env/kustomization.yaml, I added

patchesJson6902:
  - target:
      group: monitoring.coreos.com
      version: v1
      kind: PrometheusRule
      name: alerting-rules-test
    path: alerting-rules-test-patch.yaml

And the contents of the new file:

- op: replace
  path: "/spec/groups/0/rules/0/labels/priority"
  value: "MINOR"

It works after properly adding the group and version to the target. Is it possible to specify in the path the rule by its name instead of the index?

You'll need to clarify the content of each file and the patch. — Gaël J, Mar 21 '23 at 21:03
The problem is that its a nested array. You can try to use the `$patch: merge` directive, but I think in a nested array, this may not work. — The Fool, Mar 21 '23 at 23:09
@TheFool updated with jsonpatch. From what I read in the doc, it seemed like it should work, so maybe I am setting it up wrongly. — Homer, Mar 22 '23 at 07:20
Yes, after properly setting the group and version, it works. But in the patch, the path uses an index to access the first rule. Is it possible to use a name? — Homer, Mar 22 '23 at 07:26
No jsonpatch works by index only. If you want to use the name, you need to use replacements, as the replacement selector is more powerful. However, since replacements need a source, you need to create some dummy object from where you can take the value. You can omit this dummy object from the final output by adding the `config.kubernetes.io/local-config: "true"` annotation. — The Fool, Mar 22 '23 at 07:48
@TheFool Thank you for everything. One last question, is it possible to change property to all rules? For example, setting the same labels/namespace to all the rules? — Homer, Mar 22 '23 at 12:31
Jsonpatch cannot do this either. It doesn't have something like a wildcard selector. However, the replacement selector can do wildcard. — The Fool, Mar 22 '23 at 12:34

larsks · Accepted Answer · 2023-03-27T16:10:37.617

The problem here is that you're working with a custom resource, which means that Kustomize doesn't have any native knowledge of how the resource is structured. That means it will fall back on its default behavior, which is to always replace lists rather than attempting to merge them.

(NB: Your kustomization.yaml examples were using deprecated syntax; the bases keyword hasn't been necessary for a while. All the examples here use more recent syntax.)

Option 1: Custom merge strategy

For lists with an obvious merge key (like name of containers in a Pod), you can inform Kustomize about this using a custom openapi schema.

To make this work, we need to make a few changes:

We need to get the OpenAPI schema for the PrometheusRule resource.
We need to annotate the OpenAPI schema with information about how to merge the spec.groups and spec.groups.rules arrays.

This article covers these first two steps in some detail.

We need to tell Kustomize about our customized scheme using the openapi keyword:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

openapi:
  path: com.coreos.monitoring.v1.PrometheusRule.yaml

We need to ensure that the alert in our patch has a different name than the existing alert in the base resource:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: alerting-rules-test
  namespace: my-namespace
spec:
  groups:
    - name: alerting-rules-test
      interval: 60s
      rules:
        - alert: alert-testing-2
          expr: absent(application:health_microservice{service="my-service"})
          labels:
            priority: "MINOR"

With all these changes, the patch works as expected. Given a kustomization.yaml like this in overlay/env:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-namespace

openapi:
  path: com.coreos.monitoring.v1.PrometheusRule.yaml

resources:
  - ../../base/prometheus-alerts

patches:
  - path: alerting-rules-test.yaml

The output of kustomize build overlay/env is:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app-id: MY-APP-ID
    ms-id: my-ms-id
  name: alerting-rules-test
  namespace: my-namespace
spec:
  groups:
  - interval: 60s
    name: alerting-rules-test
    rules:
    - alert: alert-testing-2
      expr: absent(application:health_microservice{service="my-service"})
      labels:
        priority: MINOR
    - alert: alert-testing
      annotations:
        description: |
          Description of test alert
        summary: |
          Summary of test alert
      expr: absent(application:health_microservice{service="my-service"})
      for: 10m
      labels:
        event_id: "100"
        mc_id: ""
        namespace: my-namespace
        priority: MAJOR

I've made all the files referenced in this answer available here.

Option 2: JSONPatch

We can also make the same change using a JSONPatch resource. This doesn't require Kustomize to have any particularly knowledge of the schema. We need to add a target to the patches entry in our kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: my-namespace

resources:
  - ../../base/prometheus-alerts

patches:
  - target:
      kind: PrometheusRule
    path: alerting-rules-test.yaml

And then write our patch like this:

- op: add
  path: /spec/groups/0/rules/-
  value:
    alert: alert-testing-2
    expr: absent(application:health_microservice{service="my-service"})
    labels:
      priority: "MINOR"

This will result in output that is functionally identical to that produced by the previous method.

The customized openapi schema I used in this example is:

definitions:
  com.coreos.monitoring.v1.PrometheusRule:
    description: PrometheusRule defines recording and alerting rules for a Prometheus
      instance
    properties:
      apiVersion:
        description: 'APIVersion defines the versioned schema of this representation
          of an object. Servers should convert recognized schemas to the latest internal
          value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
        type: string
      kind:
        description: 'Kind is a string value representing the REST resource this object
          represents. Servers may infer this from the endpoint the client submits
          requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
        type: string
      metadata:
        $ref: '#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta'
        description: 'Standard object''s metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata'
      spec:
        description: Specification of desired alerting rule definitions for Prometheus.
        properties:
          groups:
            description: Content of Prometheus rule file
            x-kubernetes-patch-merge-key: name
            x-kubernetes-patch-strategy: merge
            items:
              description: RuleGroup is a list of sequentially evaluated recording
                and alerting rules.
              properties:
                interval:
                  description: Interval determines how often rules in the group are
                    evaluated.
                  pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$
                  type: string
                name:
                  description: Name of the rule group.
                  minLength: 1
                  type: string
                partial_response_strategy:
                  description: 'PartialResponseStrategy is only used by ThanosRuler
                    and will be ignored by Prometheus instances. More info: https://github.com/thanos-io/thanos/blob/main/docs/components/rule.md#partial-response'
                  pattern: ^(?i)(abort|warn)?$
                  type: string
                rules:
                  type: array
                  x-kubernetes-patch-merge-key: alert
                  x-kubernetes-patch-strategy: merge
                  description: List of alerting and recording rules.
                  items:
                    description: 'Rule describes an alerting or recording rule See
                      Prometheus documentation: [alerting](https://www.prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
                      or [recording](https://www.prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules)
                      rule'
                    properties:
                      alert:
                        description: Name of the alert. Must be a valid label value.
                          Only one of `record` and `alert` must be set.
                        type: string
                      annotations:
                        additionalProperties:
                          type: string
                        description: Annotations to add to each alert. Only valid
                          for alerting rules.
                        type: object
                      expr:
                        description: PromQL expression to evaluate.
                        x-kubernetes-int-or-string: true
                      for:
                        description: Alerts are considered firing once they have been
                          returned for this long.
                        pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$
                        type: string
                      labels:
                        additionalProperties:
                          type: string
                        description: Labels to add or overwrite.
                        type: object
                      record:
                        description: Name of the time series to output to. Must be
                          a valid metric name. Only one of `record` and `alert` must
                          be set.
                        type: string
                    required:
                      - expr
                    type: object
              required:
                - name
                - rules
              type: object
            type: array
            x-kubernetes-list-map-keys:
              - name
            x-kubernetes-list-type: map
        type: object
    required:
      - spec
    type: object
    x-kubernetes-group-version-kind:
      - group: monitoring.coreos.com
        kind: PrometheusRule
        version: v1

Kubernetes kustomize with prometheus

1 Answers1

Option 1: Custom merge strategy

Option 2: JSONPatch