Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script to wait for the model to appear when using KServe Modelcar #356

Closed
wants to merge 1 commit into from
Closed

Add script to wait for the model to appear when using KServe Modelcar #356

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ WORKDIR /caikit

COPY --from=poetry-builder /caikit/.venv /caikit/.venv
COPY caikit.yml /caikit/config/caikit.yml
COPY --chown=1001:0 --chmod=554 utils/wait-modelcar.sh .

ENV VIRTUAL_ENV=/caikit/.venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
Expand All @@ -38,4 +39,5 @@ USER caikit
ENV CONFIG_FILES=/caikit/config/caikit.yml
VOLUME ["/caikit/config/"]

ENTRYPOINT ["/caikit/wait-modelcar.sh"]
CMD ["python", "-m", "caikit.runtime"]
Comment on lines +42 to 43
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the semantics here of changing the entrypoint? Does CMD run afterwards? I was under the impression that the entrypoint is what's run when the container starts up

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When an ENTRYPOINT is specified, then the CMD becomes its arguments (the separation comes from the early Docker days, so that the image author could specify a command with ENTRYPOINT that could not be overwritten with docker run). In the Kubernetes world, ENTRYPOINT correspond to the command: field and CMD correspond to the args: field in a Pod container's spec.

14 changes: 14 additions & 0 deletions utils/wait-modelcar.sh
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this additional script really necessary? Shouldn't the storage initializer handle this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that the storage initializer runs as an init-container which runs before the modelcar sidecar starts in parallel with the runtime container.

The proper solution is to add the modelcar as a proper K8s sidecar as described in kserve/kserve#3646, but this requires K8s >= 1.29 + this feature needs to be enabled.

Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash

if [ "${MODEL_INIT_MODE}" = "async" ] ; then
echo "Waiting for model files (modelcar) to be present..."
until test -e /mnt/models; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth to also add a timeout, as there can be also error case when the modelcar can't be started. Of course, the overall Pod will be in an error state if the modelcar can't be fired up (e.g. when specifying a non-existing image as storageUrl) and then the question also would be, how large this timeout should be (really depends also on the size of the model to run, as the most likely cast when you have to wait for a model is when it is pulled from a registry).

sleep 1
done

echo "Model files are now available."
fi

echo "Starting model server..."
eval $@

Loading