Prometheus not displaying custom metrics from localhost service on `/metrics` endpoint-CodePudding

I am running a service on localhost that exports metrics to a Prometheus container running on Docker. I am able to see that my metrics are registered, scraped, and displayed correctly through the /graph endpoint of Prometheus. However, I cannot find my metrics being reported on the /metrics endpoint of Prometheus.

My service exposes metrics on localhost:8081/metrics using the Prometheus Java client. My service exposes the as follows on localhost:8081/metrics:

public class Main {
  private HTTPServer server;
  public static final int SERVER_PORT = 8081;

  /**
   * Initiates the Prometheus Exposer HTTP Server.
   */
  public void main(String args[]) {

    try {
      server = new HTTPServer(SERVER_PORT);
    } catch (IOException e) {
      throw new IllegalStateException("could not start metrics server:\t"   e);
    }

    io.prometheus.client.Gauge gauge = io.prometheus.client.Gauge
        .build()
        .namespace("some_namespace")
        .subsystem("some_subsystem")
        .name("some_name")
        .help("helpMessage")
        .register();

    int i = 0;
    while (true) {
      try {
        Thread.sleep(1000);
        gauge.set(i  );
      } catch (InterruptedException ex) {
        System.err.println("Thread sleep issue, breaking the loop");
        break;
      }
    }
  }

  /**
   * Terminates the Prometheus Exposer HTTP Server.
   */
  public void terminate() {
    try {
      server.close();
    } catch (Exception e) {
      throw new IllegalStateException("could not stop metrics server:\t"   e);
    }
  }
}

I am using the Prometheus Docker file and running it with the following configuration:

version: '2.1'
networks:
  monitor-net:
    driver: bridge
volumes:
    prometheus_data: {}
services:
  prometheus:
    image: prom/prometheus:v2.22.1
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    expose:
      - 9090
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

My prometheus.yml file is configured as follows:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'my_job'
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: [ 'host.docker.internal:8081' ]

I can see that Prometheus collects and reports several Prometheus native metrics associated with the "my_job" on the "/metrics" endpoint. However, the gauge metric that I registered through the Prometheus Java client is only accessible through the "/graph" endpoint. Here is an example of such native metrics of prometheus that is shown on "/metrics" endpoint:

# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="my_job",quantile="0.01"} 0.000311125
prometheus_target_sync_length_seconds{scrape_job="my_job"",quantile="0.05"} 0.000311125
prometheus_target_sync_length_seconds{scrape_job="my_job",quantile="0.5"} 0.000311125
prometheus_target_sync_length_seconds{scrape_job="my_job"",quantile="0.9"} 0.000311125
prometheus_target_sync_length_seconds{scrape_job="my_job"",quantile="0.99"} 0.000311125

I have also checked that:

Prometheus configuration file has the correct target and scrape configuration for my application. There were no errors or warnings related to metric collection in the logs of Prometheus and my application. Metrics are being exported in the Prometheus text format. Metrics are being exported over the correct endpoint and port that Prometheus is configured to scrape. I am still not sure why my metrics are not showing up on the "/metrics" endpoint. I will continue to investigate and troubleshoot the issue.

CodePudding user response：

Oh, you are mixing something up. http://localhost:9090/metrics is a url on your prometheus instance. It will show metrics about prometheus itself, like how much data was scraped, how much data is stored etc. It is not supposed to show the data that prometheus collected. So the metrics you found is just a status of how much time prometheus spent scraping your data.

What you actually want is to access the Prometheus API at http://localhost:9090/api/v1 and query the data about your metric. But once you understand how to run queries (and this can be done on the /graph endpoint as well) you probably want a visualization tool like Grafana.

Coming back to your question: It is working as designed.

See