Service bus scaler not scaling to more then 1 instance.

[saralpa]

This is my yaml. Pods are not scaling up more than 1. Scaling down is happening correctly

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xyz
  labels:
    app: xyz
spec:
  selector:
    matchLabels:
      app: xyz
  template:
    metadata:
      labels:
        app: xyz
    spec:
      initContainers:
        - name: mdsp-java-init
          image: aksacrname.azurecr.io/ops/mdsp-init-java:1.0
          imagePullPolicy: Always
          resources: {}
          volumeMounts:
            - name: azure
              mountPath: /mnt/azure
            - name: cert-storage
              mountPath: /certs
      volumes:
        - name: azure
          azureFile:
            shareName: certs
            secretName: mdsp-secret-iots-sa
            readOnly: true
        - name: cert-storage
          emptyDir: {}
        - name: "logging-volume-azure-file"
          persistentVolumeClaim:
            claimName: "pvc-azure-file-logging"
      containers:
        - name: xyz
          image: mdspaksdevacr.azurecr.io/iots/xyz:aksacrversion
          resources:
            limits:
              cpu: 3
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
          imagePullPolicy: Always
          env:
            - name: mdsp_product_line
              value: "iiotservices"
            - name: mdsp_application_name
              value: "ixyz"
            - name: eh_connection_string
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: LOGGING_EH_CON_STRING
            - name: log_base_path
              value: "/mnt/logging"
            - name: app.logging.console.enabled
              value: "true"
            - name: ai_instrumentation_key
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: APPINSIGHT_KEY
            - name: MDSP_HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MDSP_HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: APPLICATION_NAME
              value: "xyz"
            - name: logging.level.com.siemens.mindsphere
              value: "INFO"
            - name: PRODUCTLINE_NAME
              value: "iiotServices"
            - name: environment
              value: "azEnvironment"
            - name: CONNECTION_TIMEOUT_MS
              value: "10000"
            - name: GATEWAY_URI
              value: "https://gateway.core.mindsphere.local"
            - name: UPDATES_SERVICE_BUS_TOPIC_NAME
              value: "internal-iiot-meta-data-updates-topic"
            - name: ASSET_CLIENTID
              value: asset
            - name: AM_BASE_URL
              value: "https://gateway.core.mindsphere.local/api/assetmanagement/v3"
            - name: CORE_AUTH_TOKEN_URL
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: OAUTH_TOKEN_URI
            - name: amServiceBusConnectionString
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: advs_servicebus_conn_str
            - name: feedbackServiceBusConnectionString
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: iots_service_bus_conn_str
            - name: UPDATES_SERVICE_BUS_CONN_STR
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: iots_service_bus_conn_str
            - name: AZURE_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_ID
            - name: AZURE_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_KEY
            - name: AZURE_SUBSCRIPTION_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: SUBSCRIPTION_ID
            - name: AZURE_TENANT_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: TENANT_ID
            - name: KV_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_ID
            - name: KV_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_KEY
            - name: JAVA_OPTS
              value: >-
                -Djavax.net.ssl.trustStore=/certs/cacerts
                -Xmx2500m
            - name: KEYVAULT_URI
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOTS_KEYVAULT_URI
            - name: COSMOS_URL
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOT_COSMOS_URI
            - name: default_key
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: iots_cosmos_db_conn_key
            - name: COSMOS_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOT_COSMOS_DB_AM
            - name: COSMOS_CONTAINER_NAME
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOT_COSMOS_CON_AM
            - name: default_secret_name
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-tu
                  key: iots_bulk_data_stream
            - name: MINDSPHERE_SECURITY_OAUTH2_CLIENT_ASSET_ACCESSTOKENURI
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: OAUTH_TOKEN_URI
            - name: MINDSPHERE_SECURITY_OAUTH2_CLIENT_ASSET_ID
              value: "iottest"
          volumeMounts:
            - name: cert-storage
              mountPath: /certs
            - name: "logging-volume-azure-file"
              mountPath: "/mnt/logging"
      imagePullSecrets:
        - name: mdsp-secret-iots
      dnsPolicy: ClusterFirst
      terminationGracePeriodSeconds: 30
---

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: xyz-scaler
  labels:
    app: xyz
    deploymentName: xyz
spec:
  scaleTargetRef:
    deploymentName: xyz
  minReplicaCount: 0
  maxReplicaCount: 3
  pollingInterval: 1
  triggers:
    - type: azure-servicebus
      metadata:
        # Required
        topicName: xyz-topic
        subscriptionName: xyz-subscription
        # Optional, can use TriggerAuthentication as well
        connection: feedbackServiceBusConnectionString
        # This must be a connection string for a queue itself, and not a namespace level (e.g. RootAccessPolicy) connection string [#215](https://github.com/kedacore/keda/issues/215)
        # Optional
        messageCount: '2'
---

I am getting issue :
Command : kubectl describe scaledobject.keda.k8s.io/xyz-scaler -n iots-dev

Type Status Reason Message

---

AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric iots-dev/queueLength/&LabelSelector{MatchLabels:map[string]string{deploymentName: xyz,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get queueLength.external.metrics.k8s.io)
Events:
Type Reason Age From Message

---

Warning FailedGetExternalMetric 9s (x30 over 131m) horizontal-pod-autoscaler unable to get external metric iots-dev/queueLength/&LabelSelector{MatchLabels:map[string]string{deploymentName:xyz,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get queueLength.external.metrics.k8s.io)
Warning FailedComputeMetricsReplicas 9s (x30 over 131m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get queueLength external metric: unable to get external metric iots-dev/queueLength/&LabelSelector{MatchLabels:map[string]string{deploymentName: xyz},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get queueLength.external.metrics.k8s.io)

[zroubalik]

Please check your KEDA Metrics Server deployment, it is not running correctly

[saralpa]

I checked it is working fine. Pod is UP and running :

Kubernetes version is : 1.18

I1127 04:10:19.785479 1 main.go:77] keda_metrics_adapter "msg"="KEDA Version: 1.5.0"
I1127 04:10:19.785550 1 main.go:78] keda_metrics_adapter "msg"="KEDA Commit: 374adbe"
I1127 04:10:19.785556 1 main.go:79] keda_metrics_adapter "msg"="Go Version: go1.14.4"
I1127 04:10:19.785561 1 main.go:80] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I1127 04:10:22.689909 1 provider.go:47] keda_metrics_adapter/provider "msg"="starting"
I1127 04:10:22.689935 1 main.go:96] keda_metrics_adapter "msg"="starting adapter..."
I1127 04:10:22.915496 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1127 04:10:23.669594 1 secure_serving.go:116] Serving securely on [::]:6443

It is working fine for Service Bus Queue and blob storage

[zroubalik]

Anything suspicious in the output of: kubectl get apiservice v1beta1.external.metrics.k8s.io -oyaml?

I can see that you are using KEDA 1.5, by chance could you try the latest version 2.1? (and please refer to migration guide: https://keda.sh/docs/2.1/migration/). KEDA v1 is no longer maintained

[saralpa]

[zroubalik]

It seems like your cluster has custom-metrics-azure-apiserver deployed. Unfortunately it is using the same metrics-server apiserver extension. Therefore KEDA and this custom-metrics-azure-apiserver cannot be installed on the same cluster.

This is the reason why scaling above 1 doesn't work, it needs functional metrics server.

[saralpa]

oh ok. Could you please provide me link what I need to install. Do I need to install functional metrics server

[zroubalik]

You need to uninstall the custom-metrics-azure-apiserver which is deployed on your cluster. (you can check that in the output of kubectl get apiservice v1beta1.external.metrics.k8s.io -oyaml) and then install KEDA again, it should install KEDA metrics server and that apiservice should point to the KEDA service

[saralpa]

ok ..Let me try this

[saralpa]

It worked. Thanks a lot