How to add the scrape_configs file to Prometheus-CodePudding

I've deployed Prometheus, Alertmanager, Grafana et cetera in the cattle-monitoring-system namespace using the rancher-monitoring Helm chart. Look here and here for documentation. Everything in this namespace is running OK.

Now I'm trying to deploy an extra Prometheus instance in another namespace called indigo using the manifest files. Unfortunately the configuration scrape_configs file won't load. I checked this by running the command kubectl exec -it prometheus-indigo-0 -c prometheus -n indigo -- cat /etc/prometheus/config_out/prometheus.env.yaml. Running the same command in the other Prometheus instance will return all the scrape_configs configuration.

My deployment.yml file looks like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitoring-rancher-monitor-prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: $NAMESPACE
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: indigo
  namespace: $NAMESPACE
spec:
  serviceAccountName: prometheus
  additionalScrapeConfigs:
    name: prometheus-scrape-configs-secret
    key: prometheus-scrape-configs.yml
  resources:
    requests:
      memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-indigo
  namespace: $NAMESPACE
spec:
  type: NodePort
  ports:
  - name: web
    nodePort: 30900
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: indigo

The prometheus-scrape-configs.yml file looks like this:

global:
  evaluation_interval: 30s
  scrape_interval: 30s
scrape_configs:
- job_name: blackbox # To get metrics about the exporter itself
  metrics_path: /metrics
  static_configs:
    - targets:
      - .....

- job_name: blackbox-http # To get metrics about the exporter’s targets
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - .....
      labels:
        env: elise
    - targets:
      - .....
      labels:
        env: osb
    - targets:
      - .....
      labels:
        env: itp
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: .....

Finally, the prometheus-scrape-configs-secret is created in the Makefile:

include ../../../common.mk
include ../../environments/$(ENVIRONMENT).mk

NAMESPACE ?= indigo

.PHONY: deploy
deploy: \
    init \
    deploy-monitoring

.PHONY: init
init:

.PHONY: deploy-monitoring
deploy-monitoring:
    kubectl create secret generic prometheus-scrape-configs-secret \
    -n $(NAMESPACE) --from-file=prometheus-scrape-configs.yml

    envsubst < $(ENVIRONMENT)-deployment.yml | kubectl apply -f -

.PHONY: delete
delete: delete-monitoring

.PHONY: delete-monitoring
delete-monitoring:
    kubectl delete secret prometheus-scrape-configs-secret -n $(NAMESPACE)
    envsubst < $(ENVIRONMENT)-deployment.yml | kubectl delete -f -

So does anybody know why the scrape_configs file is unable to load correctly?

EDIT:

After running the command kubectl describe pod prometheus-indigo-0 -n indigo I noticed this error below. I also noticed after deploying this Prometheus instance it will get one error, immediately restarts and after that it is running..

State:       Running
  Started:   Tue, 14 Jun 2022 18:01:23  0200
Last State:  Terminated
  Reason:    Error
  Message:   ts=2022-06-14T16:01:19.743Z caller=main.go:450 level=error msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"

CodePudding user response：

Thanks to this documentation I managed to fix this issue. I forgot to create a ServiceMonitor that's referring to the blackbox-exporter through labels. Eventually you need to link this in the Kubernetes Prometheus object using the serviceMonitorSelector. You can look here for more information how to configure serviceMonitorSelector