When I ran the helmfile -e prd -l name=prd-alerts diff command to sync alerts to update the following expression the diff completed successfully, and showed the output that I would expect.
absent_over_time(job_run_time{job="JobName", variation="control"}[1h]) == 1
When I ran the sync command, helmfile -e prd -l name=prd-alerts sync I received the following error, which hasn't been very descriptive on how to fix whatever issue is happening.
COMBINED OUTPUT: Error: UPGRADE FAILED: cannot patch "prd-alerts" with kind PrometheusRule: admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid
I've tried many variations of updating the expression above, which included removing the logic operator ( ==1 ), removing the extra label of variation="control", tried single quotes around both labels, etc. and nothing has worked so far.
Has anyone had luck troubleshooting this error when doing a helmfile sync? I would appreciate any and all input.
