AKS nodes are not autoscaled before scheduling on ACI virtual nodes

[hasandogu]

Hello all,

I am pretty new to Kubernetes and I am seeing some rather interesting autoscaling behaviour that involves AKS nodes, virtual kubelet and KEDA. I am not sure which is the best repository this issue would lie in but I thought KEDA would be a good start for my issue.

Here is the situation:

1. We have deployed a Kubernetes cluster with virtual nodes support on AKS.

2. We are using the Azure Functions runtime in a container. Our function is using an Azure Storage Queue trigger and we used KEDA to autoscale our Kubernetes instances.

3. We have created a user nodepool set up to autoscale between 0-4 instances and observe that KEDA successfully autoscales this pool based on the outstanding messages in the queue.

4. We have also added the ability to run the pods through the virtual kubelet. We disabled our user nodepool and observed that the KEDA successfully autoscales our virtual nodes.

5. However, we would like our user nodepool instances to be used (and scaled) before the virtual kubelet instances are used during scheduling. So, we've added to our deployments the same node anti-affinity and toleration rules described here: https://github.com/Azure-Samples/virtual-node-autoscale#install-virtual-node-admission-controller-optional.

The problem is that the user nodepool scaling does not happen and the pods are scheduled directly onto the virtual nodes skipping the user nodepool autoscaling.

• If there are 0 instances currently running in our AKS user nodepool, then the pods are scheduled directly to virtual nodes

• If there is at least 1 user node currently running, then the pod gets scheduled on it first, but it will still not autoscale as the second pod that comes in will get scheduled to the virtual nodes. No new instances are added to the userpool as a result of this deployment.

How can I get more information from my cluster (or KEDA) on why the scale-up of the user nodes wouldn't happen before virtual nodes are scheduled?

Thank you all.

Dogu

AKS, KEDA, Virtual Kubelet

[tomkerkhove]

Happy to help but think this is more of an AKS question.

KEDA basically scales out the application by scheduling them, but then it's up to the Kubernetes scheduler to decide on what node to run them.

This is happening because the user nodes are not spinning up fast enough which means the workloads will flow over to Azure Container Instances through Virtual Nodes. Once the user nodes catch up, all future workloads will run there but I don't think Kubernetes will reschedule the Virtual Node pods back inside the cluster (because it doesn't know it's running outside of the cluster).

Does this make sense to you?

[hasandogu]

Thanks for the detailed explanation, @tomkerkhove .

This is happening because the user nodes are not spinning up fast enough which means the workloads will flow over to Azure Container Instances through Virtual Nodes. Once the user nodes catch up, all future workloads will run there but I don't think Kubernetes will reschedule the Virtual Node pods back inside the cluster (because it doesn't know it's running outside of the cluster).

It definitely makes sense that the future workloads will schedule to run on those nodes. However. the autoscaler doesn't seem to respond and no new nodes are spinning up at all (whether it was 0 or more at the time of scheduling). It just sits there as if no pods are pending.

Would this be more of a question to cluster autoscaler or would we go about AKS first?

Thank you for all your help.

[tomkerkhove]

I would go to the AKS repo and ask it there or open a support ticket.

[hasandogu]

Thanks @tomkerkhove . I did open Azure/AKS#2099 and @joaguas kindly provided pointers and a suggestion to try, which I will follow up with shortly.

Thank you for your help again.