The pods that successfully apply for gpu resources lack gpu card allocation information

[wangyang0616]

Description

When Volcano schedules heterogeneous resources such as GPU, it first filters the node through the predicate, and then allocates GPU resources through the callback function AllocateFunc registered by the GPU and other plug-ins.

AllocateFunc has no return value or other structure to synchronize the execution result, that is, it does not judge and process whether the GPU allocation is successful. When the GPU allocation fails, the annotation information of the GPU card allocation will not be updated on the pod. The outer layer allocate does not currently perceive the exception. Continuing to execute subsequent bind actions will cause the pod to be scheduled to the node, but it will fail to run due to the lack of corresponding GPU resources.

Steps to reproduce the issue

No response

Describe the results you received and expected

When calling the callback function AllocateFunc to allocate heterogeneous resources, the caller can perceive its execution result and process it, which is mainly divided into the following steps:

• Task1: AllocateFunc supports carrying the execution result of the callback function.
• Task2: External actions such as allocate/preempt/backfill support the processing of AllocateFunc execution exceptions.
• Task3: The gpu plug-in supports synchronizing its internal exception processing information through allocateFunc.

What version of Volcano are you using?

master

Any other relevant information

No response

[wangyang0616]

/assign