Prometheus remote_write-CodePudding

I am not very familiar with these components but I am trying. I have Grafana and Prometheus installed on an Ubuntu server on my local network. Prometheus is collecting the own Linux metrics using Node Exporter and I can display the metrics on Grafana using the Prometheus data source. So, I assume that on this side, for local use, both Prometheus and Grafana, are correctly set.

I have installed Prometheus also on a Raspberry Pi which is also on my local network. I did manage to remote write the metrics relevant the Raspberry Pi to Grafana Cloud. Now I am attempting to remote write the same metrics to the Grafana on the Ubuntu server mentioned above but I am not succeeding.

In the prometheus.yml on the Raspberry Pi, I have replaced this (which is working):

#remote_write:
#- url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push
  #name: grafana_cloud_prometheus
  #basic_auth:
    #username: something
    #password: something

with this:

remote_write:
- targets: "http://10.0.0.164/prometheus/remote/write"
  name: Prometheus
  basic_auth:
    username: admin
    password: admin

username and password are those set in Grafana.

When I restart the Prometheus service on the Pi, it fails:

Process: 8698 ExecStart=/home/pi/prometheus/prometheus --config.file=/home/pi/prometheus/prometheus.yml --storage.tsdb.path=/home/pi/prometheus/data --log.level=debug (code=e Main PID: 8698 (code=exited, status=2)

I am not sure about what I am doing wrong or if I am missing something somewhere here. I did try to manipulate the target using only the server IP, IP and port, but I am not succeeding. I would appreciate some help to get this right. Thanks

CodePudding user response：

You can't remote write to Grafana (the visualization software), it accesses data by querying Prometheus itself. Remote write is meant to be used to write to other Prometheus instances, long term storage backends (like Thanos and Cortex) or SaaS platforms (like Grafana Cloud).

What you probably want to do is add that Prometheus instance as a datasource in Grafana.

CodePudding user response：

If you need to write data from Prometheus at Raspberry PI to Prometheus at Ubuntu server, then you need to enable remote_write capability at Prometheus server running on Ubuntu - see these docs.

I'd recommend also trying to run Prometheus in Agent mode or to run vmagent instead of Prometheus in default mode at Raspberry PI. These systems allow scraping Prometheus targets in the same way as Prometheus does, but they use lower amounts of compute resources (CPU, RAM, disk space, disk IO) compared to Prometheus.