Deployed prometheus with Django and Kubernetes, how to make it scrape the Django app?-CodePudding

I have a Django project deployed in Kubernetes and I am trying to deploy Prometheus as a monitoring tool. I have successfully done all the steps needed to include django_prometheus in the project and locally I can go go localhost:9090 and play around with querying the metrics.

I have also deployed Prometheus to my Kubernetes cluster and upon running a kubectl port-forward ... on the Prometheus pod I can see some metrics of my Kubernetes resources.

Where I am a bit confused is how to make the deployed Django app metrics available on the Prometheus dashboard just like the others. I deployed my app in default namespace and prometheus in a monitoring dedicated namespace. I am wondering what am I missing here. Do I need to expose the ports on the service and deployment from 8000 to 8005 according to the number of workers or something like that?

My Django app runs with gunicorn using supervisord like so:

[program:gunicorn]
command=gunicorn --reload --timeout 200000 --workers=5 --limit-request-line 0 --limit-request-fields 32768 --limit-request-field_size 0 --chdir /code/ my_app.wsgi

my_app service:

apiVersion: v1
kind: Service
metadata:
  name: my_app
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: my-app
  sessionAffinity: None
  type: ClusterIP

Trimmed version of the deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app-deployment
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: my-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: ...
        imagePullPolicy: IfNotPresent
        name: my-app
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: regcred
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

prometheus configmap

apiVersion: v1
data:
  prometheus.rules: |-
    ... some rules
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    scrape_configs:
      - job_name: prometheus
        static_configs:
        - targets:
          - localhost:9090

      - job_name: my-app
        metrics_path: /metrics
        static_configs:
          - targets:
            - localhost:8000

      - job_name: 'node-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'node-exporter'
          action: keep

kind: ConfigMap
metadata:
  labels:
    name: prometheus-config
  name: prometheus-config
  namespace: monitoring

CodePudding user response：

You do not have to expose services, if the promehteus is installed on the same cluster as your app. You can communicate with apps between namespaces by using Kubernetes DNS resolution, going by the rule:

SERVICENAME.NAMESPACE.svc.cluster.local

so one way is to change your prometheus job target to something like this

  - job_name: speedtest-ookla
    metrics_path: /metrics
    static_configs:
      - targets:
          - 'my_app.default.svc.cluster.local:9000'

And this is the "manual" way. A better approach will be to use prometheus kubernetes_sd_config. It will autodiscover your services and try to scrape them.

Reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

CodePudding user response：

No need to expose the application outside the cluster.

Leveraging the Kubernetes service discovery, add the job to scrape Services, Pods, or both:

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    regex: ([^:] )(?::\d )?;(\d )
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (. )
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
    regex: (. )
  - regex: __meta_kubernetes_service_label_(. )
    action: labelmap
  - regex: 'app_kubernetes_io_(. )'
    action: labeldrop
  - regex: 'helm_sh_(. )'
    action: labeldrop

- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:] )(?::\d )?;(\d )
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (. )
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
    regex: (. )
  - source_labels: [__meta_kubernetes_pod_node_name]
    action: replace
    target_label: host
    regex: (. )
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod
    regex: (. )
  - regex: __meta_kubernetes_pod_label_(. )
    action: labelmap
  - regex: 'app_kubernetes_io_(. )'
    action: labeldrop
  - regex: 'helm_sh_(. )'
    action: labeldrop

Then, annotate the Service with:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "80"
    prometheus.io/path: "/metrics"

and the Deployment with:

spec:
  template:
    metadata:
     annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "80"
      prometheus.io/path: "/metrics"