Add script to wait for the model to appear when using KServe Modelcar #356
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: israel-hdez The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
When using OCI containers for model storage in KServe (modelcar), there is the possibility that the model server starts before the model has been fully downloaded. When this happens, the model server would terminate with error because the model path is empty. This adds a small script to wait for the cluster to fully download the model container before invoking the model server. The waiting is triggered when the MODEL_INIT_MODE environment variable is set to "async". Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com>
israel-hdez
force-pushed
the
modelcars-fix
branch
from
1b430e1
to
c762dc0
Compare
| ENTRYPOINT ["/caikit/wait-modelcar.sh"] | ||
| CMD ["python", "-m", "caikit.runtime"] |
There was a problem hiding this comment.
What are the semantics here of changing the entrypoint? Does CMD run afterwards? I was under the impression that the entrypoint is what's run when the container starts up
There was a problem hiding this comment.
When an ENTRYPOINT is specified, then the CMD becomes its arguments (the separation comes from the early Docker days, so that the image author could specify a command with ENTRYPOINT that could not be overwritten with docker run). In the Kubernetes world, ENTRYPOINT correspond to the command: field and CMD correspond to the args: field in a Pod container's spec.
There was a problem hiding this comment.
Is this additional script really necessary? Shouldn't the storage initializer handle this?
There was a problem hiding this comment.
The problem is that the storage initializer runs as an init-container which runs before the modelcar sidecar starts in parallel with the runtime container.
The proper solution is to add the modelcar as a proper K8s sidecar as described in kserve/kserve#3646, but this requires K8s >= 1.29 + this feature needs to be enabled.
|
|
||
| if [ "${MODEL_INIT_MODE}" = "async" ] ; then | ||
| echo "Waiting for model files (modelcar) to be present..." | ||
| until test -e /mnt/models; do |
There was a problem hiding this comment.
It might be worth to also add a timeout, as there can be also error case when the modelcar can't be started. Of course, the overall Pod will be in an error state if the modelcar can't be fired up (e.g. when specifying a non-existing image as storageUrl) and then the question also would be, how large this timeout should be (really depends also on the size of the model to run, as the most likely cast when you have to wait for a model is when it is pulled from a registry).
|
See also my comment to https://github.com/opendatahub-io/text-generation-inference/pull/98/files#r1747244274 |
|
We discussed in a call, and this won't be needed. We are changing kserve-controller to reduce the race condition. |
When using OCI containers for model storage in KServe (modelcar), there is the possibility that the model server starts before the model has been fully downloaded. When this happens, the model server would terminate with error because the model path is empty.
This adds a small script to wait for the cluster to fully download the model container before invoking the model server. The waiting is triggered when the MODEL_INIT_MODE environment variable is set to "async".
How Has This Been Tested?
Merge criteria: