Shared filesystem caching #272
Conversation
|
|
||
| set -euxo pipefail | ||
|
|
||
| huggingface-cli download --local-dir $MODEL_DIR $MODEL_REPO |
There was a problem hiding this comment.
As a follow up, we may need to do some smarter logic that ensures only required files get downloaded. For example, some repos have both safetensors and pytorch. I can tackle this.
nstogner
force-pushed
the
shared-filesystem-caching
branch
from
06873fc
to
e0ac7a2
Compare
| }, | ||
| }); err != nil { | ||
| if !apierrors.IsNotFound(err) { | ||
| return fmt.Errorf("deleting job %q: %w", jobName, err) |
There was a problem hiding this comment.
personal preference: Good scenario for errors.Join() if there is a chance that any of the next deletes could succeed.
There was a problem hiding this comment.
I cant think of many scenarios where a subsequent delete might succeed, I am going to leave as-is and allow exp-backoff of the Reconciler to take effect on this.
|
@alpe Thanks for the review! |
| Image: c.Image, | ||
| Args: args, | ||
| Env: env, | ||
| Resources: corev1.ResourceRequirements{ |
There was a problem hiding this comment.
You are not setting the SecurityContext: r.ModelServerPods.ModelContainerSecurityContext here, like on the other engines. Is there a reason for this?
| // ollama run nomic-embed-text hey | ||
| // Error: "nomic-embed-text" does not support generate | ||
| // | ||
| startupProbeScript += fmt.Sprintf(" && /bin/ollama run %s hi", m.Name) |
There was a problem hiding this comment.
I assume the purpose of this call is to verify the model is downloaded.
There is a simpler way to achieve the same:
./ollama show nomic-embed-text --modelfile
will return quicker with some metadata or fail with an error code when the server is not running or the model not available.
There was a problem hiding this comment.
Filed #277 to track to avoid scope creep
|
@alpe Really appreciate the in-depth review. I addressed most comments and opened issues for others. |
hf://urls.Scope creep:
Followup later (Issues submitted):