I have a Django project deployed in Kubernetes and I am trying to deploy Prometheus as a monitoring tool. I have successfully done all the steps needed to include django_prometheus
in the project and locally I can go go localhost:9090
and play around with querying the metrics.
I have also deployed Prometheus to my Kubernetes cluster and upon running a kubectl port-forward ...
on the Prometheus pod I can see some metrics of my Kubernetes resources.
Where I am a bit confused is how to make the deployed Django app metrics available on the Prometheus dashboard just like the others.
I deployed my app in default
namespace and prometheus in a monitoring
dedicated namespace. I am wondering what am I missing here. Do I need to expose the ports on the service and deployment from 8000 to 8005 according to the number of workers or something like that?
My Django app runs with gunicorn using supervisord
like so:
[program:gunicorn]
command=gunicorn --reload --timeout 200000 --workers=5 --limit-request-line 0 --limit-request-fields 32768 --limit-request-field_size 0 --chdir /code/ my_app.wsgi
my_app
service:
apiVersion: v1
kind: Service
metadata:
name: my_app
namespace: default
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: my-app
sessionAffinity: None
type: ClusterIP
- Trimmed version of the
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: my-app
name: my-app-deployment
namespace: default
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: my-app
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: my-app
spec:
containers:
- image: ...
imagePullPolicy: IfNotPresent
name: my-app
ports:
- containerPort: 80
name: http
protocol: TCP
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: regcred
restartPolicy: Always
schedulerName: default-scheduler
terminationGracePeriodSeconds: 30
prometheus configmap
apiVersion: v1
data:
prometheus.rules: |-
... some rules
prometheus.yml: |-
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/prometheus.rules
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- job_name: my-app
metrics_path: /metrics
static_configs:
- targets:
- localhost:8000
- job_name: 'node-exporter'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_endpoints_name]
regex: 'node-exporter'
action: keep
kind: ConfigMap
metadata:
labels:
name: prometheus-config
name: prometheus-config
namespace: monitoring
CodePudding user response:
You do not have to expose services, if the promehteus is installed on the same cluster as your app. You can communicate with apps between namespaces by using Kubernetes DNS resolution, going by the rule:
SERVICENAME.NAMESPACE.svc.cluster.local
so one way is to change your prometheus job target to something like this
- job_name: speedtest-ookla
metrics_path: /metrics
static_configs:
- targets:
- 'my_app.default.svc.cluster.local:9000'
And this is the "manual" way. A better approach will be to use prometheus kubernetes_sd_config
. It will autodiscover your services and try to scrape them.
Reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
CodePudding user response:
No need to expose the application outside the cluster.
Leveraging the Kubernetes service discovery, add the job to scrape Services, Pods, or both:
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
regex: ([^:] )(?::\d )?;(\d )
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (. )
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
regex: (. )
- regex: __meta_kubernetes_service_label_(. )
action: labelmap
- regex: 'app_kubernetes_io_(. )'
action: labeldrop
- regex: 'helm_sh_(. )'
action: labeldrop
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:] )(?::\d )?;(\d )
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (. )
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
regex: (. )
- source_labels: [__meta_kubernetes_pod_node_name]
action: replace
target_label: host
regex: (. )
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
regex: (. )
- regex: __meta_kubernetes_pod_label_(. )
action: labelmap
- regex: 'app_kubernetes_io_(. )'
action: labeldrop
- regex: 'helm_sh_(. )'
action: labeldrop
Then, annotate the Service with:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "80"
prometheus.io/path: "/metrics"
and the Deployment with:
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "80"
prometheus.io/path: "/metrics"