How to effectively monitor HPA stats for Kubernetes PODs-CodePudding

Currently we have a set of microservice hosted on kubernetes cluster. We are setting hpa values based on rough estimates. I am planning to monitor horizontal pod autoscaling behavior using grafana to ensure we are not over/under allocating the resources like CPU/memory and come up with possible cost optimization recommendation. Need directions on how to achieve this.

I am new to Kubernetes world. Need directions on how to achieve this.

CodePudding user response：

As a starting point, you could monitor CPU- and memory-consumption of each pod. For example you can do something like this:

sum by (pod) (container_memory_usage_bytes{container=...}/
sum by (pod) (kube_pod_container_resource_requests{container=...})

With such a query you can analyse, if the requested memory per pod is roughly realistic. Depending on the configuration of the autoscaler, this could be helpful.

CPU usage is also relevant:

process_cpu_usage{container="..."}

For additional queries, have a look at Prometheus queries to get CPU and Memory usage in kubernetes pods.

Now, as you have basic metrics in place, what about the autoscaler itself? You'll be able to count the number of active pods like this:

kube_horizontalpodautoscaler_status_current_replicas{}

Note that you might need to filter this metric by label horizontalpodautoscaler. But I recommend that you first run the metric without filters to get information about all running autoscalers.

A list of kubernetes metrics can be found at kube-state-metrics. Have a look at Horizontal Pod Autoscaler Metrics.

CodePudding user response：

Horizontal scaling allows you to scale up and down the number of pods depending on set metrics. Kubernetes offers Horizontal Pod Autoscaler (HPA) service to enable you to scale pods horizontally.Scaling is based on the target value. You can, for example, create metrics on CPU utilization and work to develop more pods to distribute load if CPU usage is busy.Horizontal autoscaling is based on rules that start or stop instances assigned to a resource when they reach the upper or lower limits. This helps you in cost optimization also.

The open-source Kubecost tool solves this problem by measuring detailed usage by Kubernetes concept, and correlating granular usage data with billing information from your cloud provider or cost estimates from your on-prem environment.

While the HPA is tremendously powerful, it does require significant, ongoing tuning to realize its benefits. To this end, we use Grafana dashboards for a convenient way to view HPA activity, specific to a particular pod or environment.