Autoscaling Kubernetes based on log based metrics-CodePudding

I use Autopilot on GKE. I've created some log based metrics that I'd like to use to scale up pods.

To begin with - I'm not sure if it's great idea - the metric is just number of records in DB to process... I have a feeling using logs to scale app might bring in some weird infinite loop or something....

Anyhow - I've tried entering logging.googleapis.com|user|celery-person-count as an external metric and got HPA cannot read metric value. Installed Stackdriver adapter but not too sure how to use it either.

CodePudding user response：

Yes , Horizontal Pod Autoscaling can be done using metric values. If your application is running in Kubernetes then you must use a custom metric for Autoscaling and if your application is not running in Kubernetes then you must use External metric for Autoscaling.

A custom metric can be selected for any of the following:

A particular node, Pod, or any Kubernetes object of any kind, including a CustomResourceDefinition (CRD).
The average value for a metric reported by all Pods in a Deployment.

Refer this documentation for detailed explanation of using custom metric for Horizontal Pod Autoscaling.

CodePudding user response：

Custom metrics

GitHub link:

kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole cluster-admin --user "$(gcloud config get-value account)"

resource : kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

Deploying application with metrics

git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples.git
cd kubernetes-engine-samples/custom-metrics-autoscaling/direct-to-sd

HPA example

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: custom-metric-sd
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: custom-metric-sd
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: custom-metric
      target:
        type: AverageValue
        averageValue: 20

You can check out this link for more information.

if your workload running top of VM or not on K8s use the external metrics instead of custom.

Refer to the documentation on Custom and external metrics for autoscaling workloads .

CodePudding user response：

GKE Autopilot clusters have Workload Identity enabled for consuming other GCP services, including Cloud Monitoring.

You'll want to follow the steps here in order to deploy the Custom Metrics Adapter on Autopilot clusters.

kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole cluster-admin --user "$(gcloud config get-value account)"

kubectl create namespace custom-metrics

kubectl create serviceaccount --namespace custom-metrics \
custom-metrics-stackdriver-adapter

gcloud iam service-accounts create GSA_NAME

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member "serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
    --role "roles/monitoring.viewer"

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

kubectl annotate serviceaccount \
  --namespace custom-metrics custom-metrics-stackdriver-adapter \
  iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

kubectl apply -f manifests/adapter_new_resource_model.yaml

Given that you've already deployed the adapter, you'll want to delete the deployment first, although you might just be able to run the steps starting at gcloud iam ...

You'll need to replace GSA_NAME with a name of your choosing and PROJECT_ID with your Google Cloud project ID.