Keda autoscaling not working "Client.Timeout exceeded while awaiting headers"-CodePudding

I want Keda to autoscale based on data from Prometheus. But there is "Client.Timeout exceeded while awaiting headers" error. I can succesully run example query:

http://localhost:9090/api/v1/query?query=sum(process_cpu_seconds_total)&time=2022-09-28T11:15:59Z

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1664363399,"67.67"]}]}}

This is my Keda configuration:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    name: example-deploy
  pollingInterval: 5
  cooldownPeriod: 10
  minReplicaCount: 1
  maxReplicaCount: 10

  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-main.monitoring.svc.cluster.local:9090
      metricName: process_cpu_seconds_total
      threshold: '100'
      activationThreshold: '50'
      query: sum(process_cpu_seconds_total)

But there are errors in logs:

>>> kubectl logs -n keda keda-operator-fbfcd6cd-n7q2t -c keda-operator

2022-09-28T11:30:00Z    ERROR   prometheus_scaler       error executing prometheus query        {"type": "ScaledObject", "namespace": "default", "name": "prometheus-scaledobject", "error": "Get \"http://prometheus-main.monitoring.svc.cluster.local:9090/api/v1/query?query=sum(process_cpu_seconds_total)&time=2022-09-28T11:29:57Z\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledObjectActive
        /workspace/pkg/scaling/cache/scalers_cache.go:94
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:149
2022-09-28T11:30:00Z    ERROR   scalehandler    Error getting scale decision    {"scaledobject.Name": "prometheus-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "example-deploy", "error": "Get \"http://prometheus-main.monitoring.svc.cluster.local:9090/api/v1/query?query=sum(process_cpu_seconds_total)&time=2022-09-28T11:29:57Z\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:149

>>> kubectl describe hpa keda-hpa-prometheus-scaledobject

Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23 , unavailable in v1.26 ; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                                keda-hpa-prometheus-scaledobject
Namespace:                                                           default
Labels:                                                              app.kubernetes.io/managed-by=keda-operator
                                                                     app.kubernetes.io/name=keda-hpa-prometheus-scaledobject
                                                                     app.kubernetes.io/part-of=prometheus-scaledobject
                                                                     app.kubernetes.io/version=2.8.0
                                                                     scaledobject.keda.sh/name=prometheus-scaledobject
Annotations:                                                         <none>
CreationTimestamp:                                                   Wed, 28 Sep 2022 13:29:48  0200
Reference:                                                           Deployment/example-deploy
Metrics:                                                             ( current / target )
  "s0-prometheus-process_cpu_seconds_total" (target average value):  <unknown> / 100
Min replicas:                                                        1
Max replicas:                                                        10
Deployment pods:                                                     1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric default/s0-prometheus-process_cpu_seconds_total/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: prometheus-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-process_cpu_seconds_total.external.metrics.k8s.io)
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetExternalMetric       3s (x7 over 93s)  horizontal-pod-autoscaler  unable to get external metric default/s0-prometheus-process_cpu_seconds_total/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: prometheus-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-process_cpu_seconds_total.external.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  3s (x7 over 93s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get s0-prometheus-process_cpu_seconds_total external metric: unable to get external metric default/s0-prometheus-process_cpu_seconds_total/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: prometheus-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-prometheus-process_cpu_seconds_total.external.metrics.k8s.io)

Is there a way to solve this problem?

CodePudding user response：

Answer: there was something wrong with my kubernetes setup and external metrics api was not avaible.

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq .

^ returned errors

minikube delete and minikube start solved my problem.