SandboxClaim/warm_pool usage

[manstis]

Regarding support for SandboxClaim / warm_pool:

• feat: add warm-pool provisioning for Kubernetes sandboxes #2157
• feat(kubernetes): create SandboxClaims for matching warm pools #2460

When these are complete OpenShell will have support (with the k8s driver) for a pool of Pods to act as sandboxes.

My question relates to how the pool behaves when separate Sandboxes are requested.

I can create a provider:

openshell provider create --name my-provider --type openai --credential OPENAI_API_KEY=<an-api-key>

I can create the inference for the provider:

openshell inference set \
    --provider my-provider \
    --model <a-model-name>

I can create a Sandbox template:

openshell sandbox template create openshell-warm-pool-example ...

I can finally create a sandbox:

openshell sandbox create --template openshell-warm-pool-example -- \
    curl https://inference.local/v1/chat/completions \
    --json '{"messages":[{"role":"user","content":"hello"}]}'

IIUC that ties all sandboxes from the pool to a single provider and inference configuration.

I have a need to use transient sandboxes with different provider and inference configurations.

Cold-provisioning would introduce unacceptable latency and I hoped the warm_pool would give me access to a pool of pre-provisioned hollow/empty sandboxes that I could configure with specific provider and inference configurations when instantiating a sandbox from the pool.

I just want to understand how the pool is to be used.

[manstis]

@mrunalp I see the issue was reported by yourself; IDK if you know the answers?

I cannot ping @grs the PR author here and don't want to venture outside the prescribed communication channels.

[manstis]

I wonder if whether it is as simple as:

openshell sandbox create --template openshell-warm-pool-example --provider my-provider -- \
    curl https://inference.local/v1/chat/completions \
    --json '{"messages":[{"role":"user","content":"hello"}]}'

This using a pre-warmed Pod from the Agent Sandbox pool and configuring it to use my-provider..

One way to find out is to test it when #2460 lands 👍

[grs]

You don't need to use the same providers to benefit from the same pool. What determines whether a sandbox request can claim from a warm pool is whether it matches the image, memory & cpu configuration, top level env settings (not those created for provider secrets/config).

The intention of the sandbox template is to make this relationship more clear. Templates hold the configuration that will shape the warm pool workloads (e.g. pods on kubernetes) and when you create a sandbox using a template you can choose whatever providers or policy you want, and attach/detach as needed.