I'd love to rename or drop a label from a /metrics
endpoint within my metric. The metric itself is from the kube-state-metrics
application, so nothing extraordinary. The metric looks like this:
kube_pod_container_resource_requests{container="alertmanager", instance="10.10.10.10:8080", funday_monday="blubb", job="some-kube-state-metrics", name="kube-state-metrics", namespace="monitoring", node="some-host-in-azure-1234", pod="alertmanager-main-1", resource="memory", uid="12345678-1234-1234-1234-123456789012", unit="byte"} 209715200
The label I'd love to replace is instance
because it refers to the host which runs the kube-state-metrics
application and I don't care about that. I want to have the value of node
in instance
and I've been trying so for hours now and can't find a way - I wonder if it's not possible at all!?
The way I'm grabbing the /metrics
endpoint is through the means of a scrape-config which looks as follows:
- job_name: some-kube-state-metrics
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
kubernetes_sd_configs:
- api_server: null
role: pod
scheme: http
relabel_configs:
- source_labels: [__meta_kubernetes_pod_labelpresent_kubeStateMetrics]
regex: true
action: keep
- source_labels: [__meta_kubernetes_pod_label_name]
regex: (.*)
replacement: $1
target_label: name
action: replace
- source_labels: [__meta_kubernetes_pod_container_port_name]
separator: ;
regex: http
replacement: $1
action: keep
- source_labels: [node]
regex: (.*)
replacement: blubb
target_label: funday_monday
action: replace
- action: labeldrop
regex: "unit=(.*)"
- source_labels: [ __name__ ]
regex: 'kube\_pod\_container\_resource\_requests'
action: drop
As you can tell, I've been trying to drop labels as well, namely the unit
-label (just for testing purposes) and I also tried to drop the metrics all together.
The funday_monday
is an example that changed because I wanted to know if static relabels are possible (it works!) - before it looked like this:
- source_labels: [node]
regex: (.*)
replacement: $1
target_label: funday_monday
action: replace
Help is appreciated.
CodePudding user response:
The problem is that you are doing those operations at the wrong time. relabel_configs happens before metrics are actually gathered, so, at this time, you can only manipulate the labels that you got from service discovery.
That node
label comes from the exporter. Therefore, you need to do this relabeling action under metric_relabel_configs:
metric_relabel_configs:
- source_labels: [node]
target_label: instance
Same goes for dropping metrics. If you wish a bit more info, I answered a similar question here: prometheus relabel_config drop action not working