I installed kube-prometheus-stack
via helm:
kubectl create namespace monitoring && \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && \
helm repo update && \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack
Then proceeded to deploy a fastapi application which has a metrics endpoint that Prometheus is supposed to scrape. /metrics
endpoint works fine as seen below:
from starlette_prometheus import metrics, PrometheusMiddleware
from fastapi import FastAPI, Request
app = FastAPI()
# Add Prometheus metrics as middleware
app.add_middleware(PrometheusMiddleware)
app.add_route("/metrics", metrics)
I expected both the target and service discovery to work but only service discovery appears to be working
Prometheus Operator version:
2.36.1
Kubernetes version information:
Client Version: v1.24.2
Kubernetes cluster kind:
Minikube v1.25.2-1
Here are the manifests for the deployed application:
- Namespace and deployment
apiVersion: v1
kind: Namespace
metadata:
name: code-detector-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: code-detector
namespace: code-detector-demo
labels:
release: prometheus-stack
spec:
replicas: 1
selector:
matchLabels:
app: code-detector
release: prometheus-stack
template:
metadata:
labels:
app: code-detector
release: prometheus-stack
spec:
containers:
- name: code-detector
image: <MY-IMAGE:TAG>
resources:
limits:
memory: 512Mi
cpu: 1000m
ports:
- containerPort: 8000
- Service
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
release: prometheus-stack
name: code-detector-service
namespace: code-detector-demo
spec:
selector:
app: code-detector
release: prometheus-stack
ports:
- name: code-detector-port
port: 8000
targetPort: 8000
- ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: code-detector-servicemonitor
# same namespace that Prometheus is running in
namespace: monitoring
labels:
app: code-detector
release: prometheus-stack
spec:
selector:
matchLabels:
app: code-detector
release: prometheus-stack
endpoints:
- path: /metrics
port: code-detector-port
interval: 15s
namespaceSelector:
matchNames:
- code-detector-demo # namespace where the app is running
Here's the labels in service discovery:
Targets pane don't show any picked up scraping job:
What could I be doing wrong?
CodePudding user response:
You have in your servicemonitor:
spec:
selector:
matchLabels:
app: code-detector
release: prometheus-stack
which basicaly means "get all endpoints from services with labels app=code-detector and release=prometheus-stack".
Your service don't have the label app=code-detector.