perf: imporve end to end queue processing latency

[fengruotj]

The current implementation of queue controller handles queue state operations and queuecommand operations for sequentially which may come into a performance bottleneck in performance-critical clusters. For example the end user may increase queuecommand & queue creation speed, so this pr add an option to make queue & queuecommand creation parallelly to handle such case.

Here, queue-controller workqueue pending process object metrics.

After perf: imporve end to end queue processing latency, the queue-controller workqueue pending process object finally reduced.

[volcano-sh-bot]

Welcome @fengruotj!

It looks like this is your first PR to volcano-sh/volcano.

Thank you, and welcome to Volcano. 😃

[fengruotj]

hi @kevin-wangzefeng, this is the perf: imporve end to end queue processing latency
/assign @kevin-wangzefeng

[Monokaix]

Hi, please sign off your commit with git commit -s.

[Monokaix]

/ok-to-test

[fengruotj]

Hi, please sign off your commit with git commit -s.

Hi, I've already signed off my commit wiht git commit -s. Thanks you very much~

[fengruotj]

/assign @Monokaix @kevin-wangzefeng

[Monokaix]

/lgtm
cc @hwdef @lowang-bh

[hwdef]

/lgtm

[fengruotj]

@hwdef I've considered this as well. The main issue is that the state transition of the queue involves queuecommand, and commandWorker is primarily used to handle the queuecommand CRD, which is then passed to the worker for processing by workqueue. The main performance bottleneck lies in both the worker and the commandWorker. I believe concurrency should be applied to both.

[hwdef]

@fengruotj
Yes, you are right.

[Monokaix]

/approve

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Monokaix

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Monokaix]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment