automatic labels to prometheus alertmanager rules-CodePudding

I'm using prometheus-community/prometheus chart

I'd like to add the following labels automatically to any alert manager rule firing

env=prod
cluster=project-prod-eks

so that I don't these labels manually to each alert rule.

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4. "} ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
     env: prod
     cluster: project-prod-eks              <---------------HOW to inject them?
   annotations:
     description: Too many 4XXs
     summary: More than 5% of all requests returned 4XX, this requires your attention

so that I can do something like

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4. "} ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
   annotations:
     description: Too many 4XXs on {{ $labels.env }} / {{ $labels.cluster }}  <----- THIS
     summary: More than 5% of all requests returned 4XX, this requires your attention

Any ideas?

CodePudding user response：

You can add external_labels to your prometheus.yml:

# The labels to add to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
  env: prod
  cluster: project-prod-eks

The community chart has it in values.yml:

serverFiles:
  prometheus.yml:
    external_labels:
      foo: bar
    ...

CodePudding user response：

so I did slightly differently that modifying the "serverFiles", see below

server:
  nodeSelector: 
    prometheus: "true"
  baseURL: "https://prometheus.project.io"
  enabled: true
  retention: "30d"
  strategy:
    type: RollingUpdate
  global:
    scrape_interval: 30s
    external_labels:
      env: prod
      client: client-name
      project: project-name
      cluster: project-prod-eks