HorizontalPodAutoscaler scales up pods but then terminates them instantly-CodePudding

So I have a HorizontalPodAutoscaler set up for my backend (an fpm-server and an Nginx server for a Laravel application).

The problem is that when the HPA is under load, it scales up the pods but it terminates them instantly, not even letting them get into the Running state.

The metrics are good, the scale-up behavior is as expected the only problem is that the pods get terminated right after scaling.

What could be the problem?

Edit: The same HPA is used on the frontend and it's working as expected, the problem seems to be only on the backend.

Edit 2: I have Cluster Autoscaler enabled, it does it's job, nodes get added when they are needed and then cleaned up so it's not an issue about available resources.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fpm-server
  labels:
    tier: backend
    layer: fpm
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      tier: backend
      layer: fpm
  template:
    metadata:
      labels:
        tier: backend
        layer: fpm
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}           
      containers:
        - name: fpm
          image: "{{ .Values.fpm.image.repository }}:{{ .Values.fpm.image.tag }}"
          ports:
            - name: http
              containerPort: 9000
              protocol: TCP
          env:
          {{- range $name, $value := .Values.env }}
          - name: {{ $name }}
            value: "{{ $value }}" 
          {{- end }}             
          envFrom:
            - secretRef:
                name: backend-secrets
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

hpa.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: fpm-server-hpa
  labels:
    tier: backend
    layer: fpm
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: fpm-server
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
    {{- end }}
    {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
    - type: Resource
      resource:
        name: memory
        targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
    {{- end }}

CodePudding user response：

Seems that the problem was with the replicas: {{ .Values.replicaCount }} definition. It seems that, if you are using HPA, replicas can't be used. I removed this line and the HPA started scaling.