Context

We have a Spring Boot application, deployed into K8s cluster (with 2 instances) configured with Micrometer exporter for Prometheus and visualization in Grafana.

My custom metrics

I've implemented couple of additional Micrometer metrics, that report some information regarding business data in the database (PostgreSQL) and I could see those metrics in Grafana, however separately for each pod.

Problem:

For our 2 pods in Grafana - I can see separate set of same metrics and the most recent value can be found by choosing (by label) one of the pods.

However there is no way to tell which pod reported the most recent values.

Is there a way to somehow always show the metrics values from the pod that was scraped last (ie it will contain the most fresh metric data)?

Right now in order to see the most fresh metric data - I have to switch pods and guess which one has the latest values.

(The metrics in question relate to database, therefore yielding the same values no matter the pod from which they are requested.)

CodePudding user response：

In Prometheus, you can obtain the labels of the latest scrape using topk() and timestamp() function:

topk(1,timestamp(up{job="micrometer"}))

This can then be used in Grafana to populate a (hidden) variable containing the instance name:

Name: instance
Type: Query
Query: topk(1,timestamp(up{job="micrometer"}))
Regex: /.*instance="([^"]*)".*/

I advise to active the refresh on time range change to get the last scrape in your time range.

Then you can use the variable in all your dashboard's queries:

micrometer_metric{instance="${instance}"}