Event Hub triggered Azure Function running on AKS with KEDA does not scale out-CodePudding

I have deployed an Event Hub triggered Azure Function written in Java on AKS. The function should scale out using KEDA. The function is correctly triggerd and working but it's not scaling out when the load increases. I have added sleep calls to the function implementation to make sure it's not burning through the events too fast and should be forced to scale out but this did not show any change as well.

kubectl get hpa shows the following output

NAME                   REFERENCE                TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-eventlogger   Deployment/eventlogger   64/64 (avg)   1         20        1          3m41s

This seems to be a first indicator that something is not correct as i assume the first number in the targets column is the number of unprocessed events in event hub. This stays the same no matter how many events i pump into the hub.

The Function was deployed using the following Kubernetes Deployment Manifest

data:
  AzureWebJobsStorage: <removed>
  FUNCTIONS_WORKER_RUNTIME: amF2YQ==
  EventHubConnectionString: <removed>
apiVersion: v1
kind: Secret
metadata:
  name: eventlogger
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eventlogger
  labels:
    app: eventlogger
spec:
  selector:
    matchLabels:
      app: eventlogger
  template:
    metadata:
      labels:
        app: eventlogger
    spec:
      containers:
      - name: eventlogger
        image: <removed>
        env:
        - name: AzureFunctionsJobHost__functions__0
          value: eventloggerHandler
        envFrom:
        - secretRef:
            name: eventlogger
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 240
          httpGet:
            path: /
            port: 80
            scheme: HTTP
        startupProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 240
          httpGet:
            path: /
            port: 80
            scheme: HTTP
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: eventlogger
  labels:
    app: eventlogger
spec:
  scaleTargetRef:
    name: eventlogger
  pollingInterval: 5
  cooldownPeriod: 5
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
  - type: azure-eventhub
    metadata:
      storageConnectionFromEnv: AzureWebJobsStorage
      connectionFromEnv: EventHubConnectionString
---

The Connection String of Event Hub contains the "EntityPath=" Section as described in the KEDA Event Hub Scaler Documentation and has Manage-Permissions on the Event Hub Namespace.

The output of kubectl describe ScaledObject is

Name:         eventlogger
Namespace:    default
Labels:       app=eventlogger
              scaledobject.keda.sh/name=eventlogger
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2022-04-17T10:30:36Z
  Finalizers:
    finalizer.keda.sh
  Generation:  1
  Managed Fields:
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:app:
      f:spec:
        .:
        f:cooldownPeriod:
        f:maxReplicaCount:
        f:minReplicaCount:
        f:pollingInterval:
        f:scaleTargetRef:
          .:
          f:name:
        f:triggers:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-04-17T10:30:36Z
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizer.keda.sh":
        f:labels:
          f:scaledobject.keda.sh/name:
      f:status:
        .:
        f:conditions:
        f:externalMetricNames:
        f:lastActiveTime:
        f:originalReplicaCount:
        f:scaleTargetGVKR:
          .:
          f:group:
          f:kind:
          f:resource:
          f:version:
        f:scaleTargetKind:
    Manager:         keda
    Operation:       Update
    Time:            2022-04-17T10:30:37Z
  Resource Version:  1775052
  UID:               3b6a68c1-c3b9-4cdf-b5d5-41a9721ac661
Spec:
  Cooldown Period:    5
  Max Replica Count:  20
  Min Replica Count:  0
  Polling Interval:   5
  Scale Target Ref:
    Name:  eventlogger
  Triggers:
    Metadata:
      Connection From Env:          EventHubConnectionString
      Storage Connection From Env:  AzureWebJobsStorage
    Type:                           azure-eventhub
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   False
    Type:     Ready
    Message:  Scaling is performed because triggers are active
    Reason:   ScalerActive
    Status:   True
    Type:     Active
    Status:   Unknown
    Type:     Fallback
  External Metric Names:
    s0-azure-eventhub-$Default
  Last Active Time:        2022-04-17T10:30:47Z
  Original Replica Count:  1
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type    Reason              Age   From           Message
  ----    ------              ----  ----           -------
  Normal  KEDAScalersStarted  10s   keda-operator  Started scalers watch
  Normal  ScaledObjectReady   10s   keda-operator  ScaledObject is ready for scaling

So i'm a bit stucked as i don't see any errors but it's still not behaving as expected.

Versions:

Kubernetes version: 1.21.9
KEDA Version: 2.6.1 installed using kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.6.1/keda-2.6.1.yaml
Azure Functions using Java 11 and extensionBundle in host.json is configured using version [2.8.4, 3.0.0)

CodePudding user response：

Was able to find a solution to the problem.

Event Hub triggered Azure Functions deployed on AKS show the same scaling characteristics as Azure Functions on App Service show:

You only get one consumer per partition to allow for ordering per partition. This characteristic overrules maxReplicaCount in the Kubernetes Deployment Manifest.

So to solve my own issue: By increasing the partitions for Event Hub i get a pod per partition and KEDA scales the workload as expected.