feat: add hierarchical queues for capacity plugin #3743

Rui-Gan · 2024-09-23T04:53:08Z

No description provided.

hwdef · 2024-09-23T06:14:47Z

ref: #3590

hwdef · 2024-09-23T06:15:47Z

/assign @Monokaix @lowang-bh @shinytang6
Please take a look 👀

pkg/scheduler/plugins/capacity/capacity.go

pkg/webhooks/admission/queues/mutate/mutate_hierarchical_queue.go

pkg/webhooks/admission/queues/validate/validate_hierarchical_queue.go

Monokaix · 2024-09-24T09:18:41Z

Is somewhere validate that job can only be submitted to leaf node?

pkg/scheduler/framework/session_plugins.go

pkg/scheduler/plugins/capacity/capacity.go

hwdef · 2024-09-26T10:44:22Z

Please add some E2E test for this function. Because this function is very important.

pkg/webhooks/admission/queues/mutate/mutate_hierarchical_queue.go

Rui-Gan · 2024-09-29T15:52:04Z

Please add some E2E test for this function. Because this function is very important.

I will add it later

Rui-Gan · 2024-09-29T16:03:29Z

Is somewhere validate that job can only be submitted to leaf node?

I have already added this logic in the validate webhook of the job to ensure that jobs can only be submitted to leaf queues.

pkg/webhooks/admission/jobs/validate/admit_job.go

googs1025 · 2024-10-03T07:47:45Z

/cc

hwdef · 2024-10-03T14:19:48Z

Please consider this scenario:

Use default config create a cluster (./hack/local-up-volcano.sh)
This configuration uses the proportion plugin by default and automatically creates a default queue
Switch to the capacity plugin and enable hierarchical queues

I believe this is how most existing users use hierarchical queues.

result:

root queue will not be created.
scheduler panic:

2024/10/03 14:13:01 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
I1003 14:13:01.520160       1 flags.go:57] FLAG: --add-dir-header="false"
I1003 14:13:01.520174       1 flags.go:57] FLAG: --alsologtostderr="false"
I1003 14:13:01.520176       1 flags.go:57] FLAG: --ca-cert-file=""
I1003 14:13:01.520179       1 flags.go:57] FLAG: --cache-dump-dir="/tmp"
I1003 14:13:01.520181       1 flags.go:57] FLAG: --cache-dumper="true"
I1003 14:13:01.520183       1 flags.go:57] FLAG: --csi-storage="false"
I1003 14:13:01.520185       1 flags.go:57] FLAG: --default-queue="default"
I1003 14:13:01.520187       1 flags.go:57] FLAG: --enable-healthz="true"
I1003 14:13:01.520189       1 flags.go:57] FLAG: --enable-metrics="true"
I1003 14:13:01.520192       1 flags.go:57] FLAG: --feature-gates=""
I1003 14:13:01.520196       1 flags.go:57] FLAG: --healthz-address=":11251"
I1003 14:13:01.520199       1 flags.go:57] FLAG: --ignored-provisioners="[]"
I1003 14:13:01.520210       1 flags.go:57] FLAG: --kube-api-burst="2000"
I1003 14:13:01.520214       1 flags.go:57] FLAG: --kube-api-qps="2000"
I1003 14:13:01.520218       1 flags.go:57] FLAG: --kubeconfig=""
I1003 14:13:01.520221       1 flags.go:57] FLAG: --leader-elect="false"
I1003 14:13:01.520223       1 flags.go:57] FLAG: --leader-elect-lease-duration="15s"
I1003 14:13:01.520229       1 flags.go:57] FLAG: --leader-elect-renew-deadline="10s"
I1003 14:13:01.520232       1 flags.go:57] FLAG: --leader-elect-resource-lock="leases"
I1003 14:13:01.520234       1 flags.go:57] FLAG: --leader-elect-resource-name="volcano"
I1003 14:13:01.520237       1 flags.go:57] FLAG: --leader-elect-resource-namespace="volcano-system"
I1003 14:13:01.520239       1 flags.go:57] FLAG: --leader-elect-retry-period="2s"
I1003 14:13:01.520242       1 flags.go:57] FLAG: --listen-address=":8080"
I1003 14:13:01.520244       1 flags.go:57] FLAG: --lock-object-namespace="volcano-system"
I1003 14:13:01.520247       1 flags.go:57] FLAG: --log-backtrace-at=":0"
I1003 14:13:01.520251       1 flags.go:57] FLAG: --log-dir=""
I1003 14:13:01.520255       1 flags.go:57] FLAG: --log-file=""
I1003 14:13:01.520259       1 flags.go:57] FLAG: --log-file-max-size="1800"
I1003 14:13:01.520263       1 flags.go:57] FLAG: --log-flush-frequency="5s"
I1003 14:13:01.520267       1 flags.go:57] FLAG: --logtostderr="true"
I1003 14:13:01.520270       1 flags.go:57] FLAG: --master=""
I1003 14:13:01.520274       1 flags.go:57] FLAG: --minimum-feasible-nodes="100"
I1003 14:13:01.520284       1 flags.go:57] FLAG: --minimum-percentage-nodes-to-find="5"
I1003 14:13:01.520287       1 flags.go:57] FLAG: --node-selector="[]"
I1003 14:13:01.520295       1 flags.go:57] FLAG: --node-worker-threads="20"
I1003 14:13:01.520303       1 flags.go:57] FLAG: --one-output="false"
I1003 14:13:01.520306       1 flags.go:57] FLAG: --percentage-nodes-to-find="0"
I1003 14:13:01.520309       1 flags.go:57] FLAG: --plugins-dir=""
I1003 14:13:01.520312       1 flags.go:57] FLAG: --priority-class="true"
I1003 14:13:01.520315       1 flags.go:57] FLAG: --schedule-period="1s"
I1003 14:13:01.520319       1 flags.go:57] FLAG: --scheduler-conf="/volcano.scheduler/volcano-scheduler.conf"
I1003 14:13:01.520322       1 flags.go:57] FLAG: --scheduler-name="[volcano]"
I1003 14:13:01.520328       1 flags.go:57] FLAG: --skip-headers="false"
I1003 14:13:01.520331       1 flags.go:57] FLAG: --skip-log-headers="false"
I1003 14:13:01.520333       1 flags.go:57] FLAG: --stderrthreshold="2"
I1003 14:13:01.520338       1 flags.go:57] FLAG: --tls-cert-file=""
I1003 14:13:01.520340       1 flags.go:57] FLAG: --tls-private-key-file=""
I1003 14:13:01.520343       1 flags.go:57] FLAG: --v="3"
I1003 14:13:01.520348       1 flags.go:57] FLAG: --version="false"
I1003 14:13:01.520350       1 flags.go:57] FLAG: --vmodule=""
W1003 14:13:01.520367       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.


panic: failed init default queue, with err: admission webhook "mutatequeue.volcano.sh" denied the request: failed to get parent queue of open queue default: queues.scheduling.volcano.sh "root" not found

goroutine 1 [running]:
volcano.sh/volcano/pkg/scheduler/cache.newDefaultQueue({0x25de198, 0xc0000e3620}, {0x22a7c80, 0x7})
        /go/src/volcano.sh/volcano/pkg/scheduler/cache/cache.go:513 +0x1d3
volcano.sh/volcano/pkg/scheduler/cache.newSchedulerCache(0xc00017d688, {0xc00005aad0, 0x1, 0x1}, {0x22a7c80, 0x7}, {0x0, 0x0, 0x0}, 0x14, ...)
        /go/src/volcano.sh/volcano/pkg/scheduler/cache/cache.go:532 +0xfd
volcano.sh/volcano/pkg/scheduler/cache.New(...)
        /go/src/volcano.sh/volcano/pkg/scheduler/cache/cache.go:92
volcano.sh/volcano/pkg/scheduler.NewScheduler(0xc00017d688?, 0xc0000ec000)
        /go/src/volcano.sh/volcano/pkg/scheduler/scheduler.go:70 +0xeb
volcano.sh/volcano/cmd/scheduler/app.Run(0xc0000ec000)
        /go/src/volcano.sh/volcano/cmd/scheduler/app/server.go:71 +0x1a5
main.main()
        /go/src/volcano.sh/volcano/cmd/scheduler/main.go:86 +0x325

installer/helm/chart/volcano/config/volcano-admission.conf

pkg/scheduler/plugins/capacity/capacity.go

pkg/webhooks/admission/queues/validate/validate_queue.go

pkg/scheduler/cache/cache.go

pkg/scheduler/api/types.go

pkg/scheduler/framework/session_plugins.go

TaiPark · 2024-10-15T06:55:47Z

In the scenario of scheduling directly with PodGroup (without Volcano Job), the logic for checking whether a Parent Queue is unable to submit is missing.

Maybe we need a webhooks.admission.podgroups.validate to check this?

Rui-Gan · 2024-10-15T07:05:02Z

In the scenario of scheduling directly with PodGroup (without Volcano Job), the logic for checking whether a Parent Queue is unable to submit is missing.

Maybe we need a webhooks.admission.podgroups.validate to check this?

Thank you for your suggestion. I will add the missing logic.

volcano-sh-bot · 2024-10-15T13:49:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign lowang-bh
You can assign the PR to them by writing /assign @lowang-bh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/controllers/queue/queue_controller_action.go

pkg/scheduler/framework/session_plugins.go

pkg/scheduler/cache/cache.go

pkg/webhooks/admission/jobs/validate/admit_job.go

hwdef · 2024-10-23T02:46:27Z

Look good to me.
@Monokaix @googs1025 @TaiPark @lowang-bh Please review this again. Can we merge this PR?

@Rui-Gan please squash your commit to one. and rebase master code

Signed-off-by: Rui-Gan <ganrui.cs@gmail.com>

pkg/scheduler/cache/cache.go

TaiPark · 2024-10-23T12:39:06Z

In the scenario of scheduling directly with PodGroup (without Volcano Job), the logic for checking whether a Parent Queue is unable to submit is missing.
Maybe we need a webhooks.admission.podgroups.validate to check this?

Thank you for your suggestion. I will add the missing logic.

Is this proposal still under consideration? I don't seem to find related logics.

Rui-Gan · 2024-10-23T13:04:08Z

In the scenario of scheduling directly with PodGroup (without Volcano Job), the logic for checking whether a Parent Queue is unable to submit is missing.
Maybe we need a webhooks.admission.podgroups.validate to check this?

Thank you for your suggestion. I will add the missing logic.

Is this proposal still under consideration? I don't seem to find related logics.

Regarding this issue, I think it would be better to submit another PR to handle this after the current PR is merged. There are a couple of reasons for this:

The use case of directly using podgroup for scheduling is relatively rare, which is why there was no related validate webhook before. For example, when creating or updating the queue field of a podgroup, there is no validation to check if the queue exists. The hdrf plugin also does not perform any validation related to hierarchical queue enablement.
Currently, adding this validation logic involves a significant amount of code changes, and the e2e tests cannot pass successfully. This is because the deployment yaml files need to be modified, and when running e2e tests on GitHub, the old yaml files are used. Therefore, if we directly include this in the current PR, the e2e tests will not pass.

Based on the above considerations, I believe this logic can be implemented in a separate PR after the current PR is merged. What do you think? Is this a particularly urgent matter?

TaiPark · 2024-10-24T03:17:03Z

In the scenario of scheduling directly with PodGroup (without Volcano Job), the logic for checking whether a Parent Queue is unable to submit is missing.
Maybe we need a webhooks.admission.podgroups.validate to check this?

Thank you for your suggestion. I will add the missing logic.

Is this proposal still under consideration? I don't seem to find related logics.

Regarding this issue, I think it would be better to submit another PR to handle this after the current PR is merged. There are a couple of reasons for this:

The use case of directly using podgroup for scheduling is relatively rare, which is why there was no related validate webhook before. For example, when creating or updating the queue field of a podgroup, there is no validation to check if the queue exists. The hdrf plugin also does not perform any validation related to hierarchical queue enablement.

Currently, adding this validation logic involves a significant amount of code changes, and the e2e tests cannot pass successfully. This is because the deployment yaml files need to be modified, and when running e2e tests on GitHub, the old yaml files are used. Therefore, if we directly include this in the current PR, the e2e tests will not pass.

Based on the above considerations, I believe this logic can be implemented in a separate PR after the current PR is merged. What do you think? Is this a particularly urgent matter?

If the implementation is quite complex, it would be acceptable to commit a separate PR for it.

hwdef · 2024-10-24T07:05:44Z

/lgtm

TaiPark · 2024-10-24T07:34:53Z

I don't have any more questions, looks good to me

volcano-sh-bot requested review from hudson741, shinytang6 and william-wang September 23, 2024 04:53

volcano-sh-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 23, 2024

Rui-Gan mentioned this pull request Sep 23, 2024

Support Hierarchical Queue on Capacity Plugin #3590

Open

Rui-Gan force-pushed the gr/queue branch from 616b098 to 32dbd06 Compare September 23, 2024 05:11

volcano-sh-bot assigned lowang-bh, Monokaix and shinytang6 Sep 23, 2024