Autoscaling jobs with KEDA - processing mulitiple jobs at once.

[sai25590]

Our use-case is as and when messages come off a service bus we have to scale the pods. For 1 message in queue we want 1 pod with a maximum of 80 pods running in parallel.

I used scaled object and scaling a job inorder to do it. The pods are configured such that if no message comes for 10 minutes it exits successfully. The parallelism and completions are 80 with min replicas 0 and max replica 100. (I don't know when and how the min and max replica is used for). The problem is that the scaling is not how we would like it to be.

1 message comes in the queue. It spins up 1 job which in turn spins up 80 pods. One of the 80 pods picks up the message. Messages are usually long running operations. It takes like 1 hour to finish. After 10 minutes the rest of the 79 pods exit successfully since there are no message to perform. LEt's say on the 11th minute 10 more messages come on the queue. The problem is all the 10 messages wait for 1 hours for this 1 pod to complete. Looks like jobs are not parallel. How to make them parallel? I thought maxreplicas was the setting but that doesn't work.

My current scaled object setting:

---
# Example followed: https://github.com/ross-p-smith/Mercury/blob/2eb97992007b104581cf1381eed11d93eb950bd8/charts/kedaV1/templates/test-scaledobject.yaml
# https://keda.sh/docs/1.4/concepts/scaling-jobs/
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
namespace: default # must be same namespace as the Triggerauth
metadata:
  name: autofab-configurator-scaledobject
spec:
  scaleType: job
  jobTargetRef:
    parallelism: ${maxconfiguratorparallelism} # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: ${maxconfiguratorparallelism} # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6
    template:
      # describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)
      metadata:
        labels:
          app: autofab-configurator
          aadpodidbinding: azureresourceaccess
        name: autofab-configurator
      name: autofab-configurator
      spec:   
        containers:
          - name: autofab-configurator
            image: ${acrlogin}/${autofabconfiguratorimage}:${version}
            imagePullPolicy: Always
            resources:
              requests:
                cpu: 100m
            volumeMounts:
              - mountPath: /mnt/azure
                name: volume
            env:
              - name: SERVICEBUS_NAMESPACE
                value: ${sbnamespace}
              - name: CONFIG_QUEUE_NAME
                value: "autofab-q"
              - name: KEYVAULT_NAME
                value: ${kvname}
      restartPolicy: Never
      volumes:
        - name: volume
          persistentVolumeClaim:
            claimName: azurefile
      
  pollingInterval: 30  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 0   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName and namespace OR topicName and subscriptionName
      queueName: autofab-q
      namespace: ${sbnamespace}
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages
    authenticationRef:
      name: autoscale-servicebus-triggerauth

Any help is appreciated :)

[sai25590]

Update: I understood what the min and max replicas are for. It's for scaling the pods (and not jobs). So, how to run multiple jobs in parallel using KEDA ScaledObject?

[sai25590]

Another update is that even though the pods don't exit the pods are are terminated after 10 minutes. I am guessing this is from this setting:

activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer

D:\git\Networking-Tycoon\src\AVS2.0-AutoFabric>kubectl get pods -l app=autofab-configurator --watch
NAME                                            READY   STATUS              RESTARTS   AGE
autofab-configurator-scaledobject-5hndj-269rf   0/1     ContainerCreating   0          27s
autofab-configurator-scaledobject-5hndj-69trn   0/1     ContainerCreating   0          27s
autofab-configurator-scaledobject-5hndj-h25fg   0/1     ContainerCreating   0          27s
autofab-configurator-scaledobject-5hndj-prrq9   0/1     ContainerCreating   0          27s
autofab-configurator-scaledobject-5hndj-vtzws   0/1     ContainerCreating   0          27s
autofab-configurator-scaledobject-5hndj-h25fg   1/1     Running             0          48s
autofab-configurator-scaledobject-5hndj-vtzws   1/1     Running             0          49s
autofab-configurator-scaledobject-5hndj-269rf   1/1     Running             0          49s
autofab-configurator-scaledobject-5hndj-69trn   1/1     Running             0          49s
autofab-configurator-scaledobject-5hndj-prrq9   1/1     Running             0          49s
autofab-configurator-scaledobject-5hndj-69trn   1/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-269rf   1/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-prrq9   1/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-h25fg   1/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-vtzws   1/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-69trn   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-prrq9   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-vtzws   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-h25fg   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-269rf   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-h25fg   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-h25fg   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-prrq9   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-prrq9   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-69trn   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-69trn   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-vtzws   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-vtzws   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-269rf   0/1     Terminating         0          10m
autofab-configurator-scaledobject-5hndj-269rf   0/1     Terminating         0          10m

I don't want KEDA to terminate the pod before it exits. Not sure how to make it do what I like.

[sai25590]

Basically this page https://keda.sh/docs/1.4/concepts/scaling-jobs/ says that :
4. As additional messages arrive, additional jobs are created. Each job processes a single message to completion.

But really when a job is being processed additional jobs aren't created by KEDA. How to make it create additional jobs?

[zroubalik]

We have just released KEDA v2 Beta, scaling of jobs was rewritten in this release. I'd recommend you to take a look on it:

https://keda.sh/blog/keda-2.0-beta/

[sai25590]

Thanks with KEDA 2.0 things work great.