Grafana Datasource Prometheus 404-CodePudding

We are using AWS EKS, i deployed Promethus using the below command:

kubectl create namespace prometheus

helm install prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"

Once this is done i get this message: The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:

The services on my prometheus deployment looks like blow:

NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/prometheus-alertmanager         ClusterIP   10.22.210.131   <none>        80/TCP     20h
service/prometheus-kube-state-metrics   ClusterIP   10.12.43.248    <none>        8080/TCP   20h
service/prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP   20h
service/prometheus-pushgateway          ClusterIP   10.130.54.42     <none>        9091/TCP   20h
service/prometheus-server               ClusterIP   10.90.94.70      <none>        80/TCP     20h

I am now using this URL in the datasource on Grafana as:

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-alertmanager.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

Grafana is also up, but when the default datasource which is prometheus in this case is unable to pull any data, when i check in the datasources tab on Grafana and try to test the datasource i am getting Error reading Prometheus: client_error: client error: 404

since both these deployments are on the same cluster ideally it should have been able to access this. Any help here would be highly appreciated.

CodePudding user response：

This is because you're targeting the wrong service. You're using the alert manager url instead of the prometheus server.
The URL should be this one :

url: http://prometheus-server.prometheus.svc.cluster.local