1

I have a scenario where I need to send the current time stamp from Prometheus to Pagerduty. I need to access the current time stamp from the Custom details in Pagerduty to write Event Rules based on the current time.

Currently I can see three fields inside Custom Details in Pagerduty ["firing", "num_firing", "num_resolved"]. All the alert related details are inside "firing" field but as a single string.

I thought of adding the time stamp to the Labels within the Prometheus event rules, but the problem is that I will not be able to retrieve that value from the "firing" field as it is not in a structured format.

Event details in Pagerduty received from Prometheus:

{
 "client": "AlertManager",
 "client_url": "http://********/",
 "description": "[FIRING:1] **************",
 "event_type": "trigger",
 "incident_key": "********",
 "service_key": "********",
 "details": {
  "firing": "Labels:\n - alertname = ******\n - datacenter = *****\n - instance = ******\n - instance_id = ******\n - instance_type = ******\n - job = ******\n - metrics = ******\n - node = ******\n - pod = ******\n - private_ip = ******\n - public_ip = ******\n - service = ******\n - severity = critical\nAnnotations:\n - description = ******\n - summary = ******\nSource: ******\n",
  "num_firing": "1",
  "num_resolved": "0",
  "resolved": ""
 }
}

I need to add the current time stamp in the same level as the "firing" field. Is there any way to do this via Prometheus alert manager configuration or via the Alert rules?

user51
  • 8,843
  • 21
  • 79
  • 158
Sujai Sivasamy
  • 1,101
  • 3
  • 14
  • 32

3 Answers3

2

The standard prometheus way for adding additional metadata to alerts is through annotations.

- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ query "time()" }}

This should add the timestamp annotation to the alert being fired.

yamenk
  • 46,736
  • 10
  • 93
  • 87
  • Hi, Thanks for your reply. I tried this (timestamp: {{ time() }}) and I am getting an error like " cannot unmarshal !!map into string". I enclosed it with double quotes like (timestamp: "{{ time() }}" ) and still getting an error like "function "time" not defined" – Sujai Sivasamy Nov 27 '18 at 06:57
  • @SujaiSivasamy updated answer which was missing the `query` function that is needed to evaluate PromQL expressions in annotations. – yamenk Nov 27 '18 at 10:36
  • still I am getting the same error as "cannot unmarshal !!map into string" while running the promtool check rules command. Is there anything to do with the versions that we use? – Sujai Sivasamy Nov 27 '18 at 10:49
  • It passes the promtool validation now. But the value that I am getting for the query is "timestamp = time: [0xc46b3c7ec0]". Any idea regarding what went wrong? – Sujai Sivasamy Nov 28 '18 at 09:42
  • Hey, I found that it's an hexadecimal value. When I converted it into decimal and to the human readable time via epoch converter, it gave me the time as "GMT: Sunday, September 15, 1996 5:51:24.928 PM", which is wrong. I am not sure what I missed here. – Sujai Sivasamy Nov 29 '18 at 08:36
2
- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }} {{ end }}

I have solved same problem, as above. Cheers.!

Chintaman
  • 91
  • 2
  • 6
1

You can directly add this as a custom detail to the alertmanager.yaml .

details:
  time: {{ query "time()" }}

Note that this adds the time from alertmanager's point of view so might contain a small delay.

Sutirtha Das
  • 163
  • 8