{{ define "default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts.Firing -}}
{{- if eq $index 0 -}}
Detail: {{ $alert.Labels.alertname }}
# Here I'm going to zero $alert
{{- end }}
{{- end }}
{{- end }}
{{- end }}
hello, I want to zero .Alerts.Firing
, which is also $alert
after iteration over it.
I'm new to golang, Please help, thanks.
Supplement:
Actually it's a template of Prometheus
alert. The orginal date is created by alertmanager
and send to receiver
( wechat).
Here is wechat template
{{ define "wechat.default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts.Firing -}}
{{- if eq $index 0 -}}
Detail: {{ $alert.Labels.alertname }}
{{- end }}
---- Problem started ------
Message: {{ $alert.Annotations.description }}
Time: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{ if gt (len $alert.Labels.instance) 0 }}ip: {{ $alert.Labels.instance }};{{- end }}
{{- if gt (len $alert.Labels.namespace) 0 }}namespace: {{ $alert.Labels.namespace }};{{- end }}
{{- if gt (len $alert.Labels.node) 0 }}Node: {{ $alert.Labels.node }};{{- end }}
{{- if gt (len $alert.Labels.pod_name) 0 }}Pod: {{ $alert.Labels.pod_name }}{{- end }}
-----------------
{{ $alert :="" }}
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts.Resolved -}}
{{- if eq $index 0 }}
!!!! Problem resolved !!!!!
Detail: {{ $alert.Labels.alertname }}
{{- end }}
Started_at: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
Resolved_at: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
-------
{{ if gt (len $alert.Labels.instance) 0 -}}ip: {{ $alert.Labels.instance }};{{- end }}
{{- if gt (len $alert.Labels.namespace) 0 -}}namespace: {{ $alert.Labels.namespace }};{{- end }}
{{- if gt (len $alert.Labels.node) 0 -}}Node: {{ $alert.Labels.node }};{{- end }}
{{- if gt (len $alert.Labels.pod_name) 0 -}}Pod: {{ $alert.Labels.pod_name }}{{- end }}
{{ $alert :="" }}
-----------------
{{- end }}
{{- end }}
{{- end }}
So it's hard to see the raw data sent by alertmanager
. What I can see only it the result rendered by this template. Please see below.
################# The first alert #########
##### As we can see 232.20 is in the state of 'resolved' ###
Detail: Node is down
---- Problem started ------
Message: Node is down
VALUE = 0
LABELS = 172.16.232.18:9100
Time: 2022-06-02 14:47:51
ip: 172.16.232.18:9100;
-----------------
!!!! Problem resolved !!!!!
Detail: Node is down
Started_at: 2022-06-02 14:47:21
Resolved_at: 2022-06-02 14:47:21
-------
ip: 172.16.232.20:9100;
-----------------
##################### The second alert #######
####### 232.20 is still in the state of resolved, it seems it's duplicated, that's why I'm going to zero it ####
!!!! Problem resolved !!!!!
Detail: Node is down
Started_at: 2022-06-02 14:47:51
Resolved_at: 2022-06-02 14:47:51
-------
ip: 172.16.232.18:9100;
-----------------
Started_at: 2022-06-02 14:47:21
Resolved_at: 2022-06-02 14:47:21
-------
ip: 172.16.232.20:9100;
-----------------
Supplyment:
- raw data of first alert.
[FIRING:1] Node is down (powerdns prod node_exporter critical)
instance is down.
Alerts Firing:
Labels:
- alertname = Node is down
- app = powerdns
- env = prod
- instance = 172.16.232.18:9100
- job = node_exporter
- severity = critical
Annotations:
- description = Node is down
VALUE = 0
LABELS = 172.16.232.18:9100
- summary = instance is down.
Source: http://pt-prometheus-server-85944ccff6-qvtc4:9090/graph?g0.expr=up{job="node_exporter"} == 0&g0.tab=1
Alerts Resolved:
Labels:
- alertname = Node is down
- app = powerdns
- env = prod
- instance = 172.16.232.20:9100
- job = node_exporter
- severity = critical
Annotations:
- description = Node is down
VALUE = 0
LABELS = 172.16.232.20:9100
- summary = instance is down.
Source: http://pt-prometheus-server-85944ccff6-qvtc4:9090/graph?g0.expr=up{job="node_exporter"} == 0&g0.tab=1
AlertmanagerUrl:
http://alertmgr.pacific-textiles.com/#/alerts?receiver=wechat
and
2. raw data of second alert.
[RESOLVED] Node is down (powerdns prod node_exporter critical)
instance is down.
Alerts Resolved:
Labels:
- alertname = Node is down
- app = powerdns
- env = prod
- instance = 172.16.232.18:9100
- job = node_exporter
- severity = critical
Annotations:
- description = Node is down
VALUE = 0
LABELS = 172.16.232.18:9100
- summary = instance is down.
Source: http://pt-prometheus-server-85944ccff6-qvtc4:9090/graph?g0.expr=up{job="node_exporter"} == 0&g0.tab=1
Labels:
- alertname = Node is down
- app = powerdns
- env = prod
- instance = 172.16.232.20:9100
- job = node_exporter
- severity = critical
Annotations:
- description = Node is down
VALUE = 0
LABELS = 172.16.232.20:9100
- summary = instance is down.
Source: http://pt-prometheus-server-85944ccff6-qvtc4:9090/graph?g0.expr=up{job="node_exporter"} == 0&g0.tab=1
AlertmanagerUrl:
http://alertmgr.pacific-textiles.com/#/alerts?receiver=wechat
CodePudding user response:
Just relealize it cannot be resolved by zero an object. Because:
- There is no way to keep last 'resolved' host 232.20 in wechat.
- Next time the raw data still contains 'resolved' host 232.20, so it cannot be removed by any means.
- so the output cannot be changed, either.
Thanks.