Skip to content

fix(serverless): default workersStandby to workersMin so workers=(0, N) scales to zero - #372

Draft
justinwlin wants to merge 1 commit into
mainfrom
fix/364-standby-follows-workers-min
Draft

fix(serverless): default workersStandby to workersMin so workers=(0, N) scales to zero#372
justinwlin wants to merge 1 commit into
mainfrom
fix/364-standby-follows-workers-min

Conversation

@justinwlin

Copy link
Copy Markdown

Summary

workers=(min, max) did not set workersStandby when building the saveEndpoint create/update payload — the field was simply absent from the model, so the RunPod API applied its own server-side default. Deploying with workers=(0, 1) produced workersMin: 0 and workersStandby: 1: the endpoint kept one warm worker forever and never scaled to zero, invisibly. This PR defaults workersStandby to workersMin (with explicit-override support), so the deployed config matches the requested scaling range.

Fixes #364

Internal: CON-1229

What changed

  • src/runpod_flash/core/resources/serverless.py — new optional workersStandby field on ServerlessResource; a model validator defaults it to workersMin when unset and validates explicit values against the (workersMin, workersMax) range. Because the deploy/update payloads are built from model_dump(), both create and update now send workersStandby.
  • src/runpod_flash/endpoint.py — new keyword-only workers_standby param on Endpoint, range-validated and passed through to the resource config (issue suggestion ux update to terminal #2). When omitted, the resource default (workersMin) applies.
  • docs/Flash_SDK_Reference.md — document workers_standby and the standby-follows-min behavior next to workers.
  • tests/unit/resources/test_serverless.py, tests/unit/test_endpoint.py — 12 new tests: workers=(0, 1) yields workersMin: 0 / workersStandby: 0 in the deploy payload, workers=(2, 5) yields workersStandby: 2, explicit overrides respected, out-of-range values raise.

Notes

  • workersStandby is intentionally not in _hashed_fields, so resource_id identity is unchanged. It does enter the exclude-based config_hash; for legacy pickled resources the field is simply absent (verified config_hash/model_dump don't error), so the first deploy after upgrade detects drift once and issues a non-structural saveEndpoint update that corrects standby on existing endpoints.
  • LB resources keep their existing behavior via workersMin=1 defaults.

How verified

  • pytest tests/unit/test_endpoint.py tests/unit/resources/test_serverless.py tests/unit/resources/test_live_serverless.py — 342 passed.
  • Full pytest tests/unit — 2570 passed, 3 failed; the same 3 tests fail identically at the base commit (unrelated pre-existing test-ordering issues in test_load_balancer_sls_stub.py / test_regressions.py; they pass in isolation).
  • ruff format + ruff check — clean. mypy . — same 260 pre-existing errors as the base commit (error set identical modulo line shifts).
  • Simulated a legacy unpickled ServerlessResource (no workersStandby in instance state): config_hash, resource_id, and model_dump all work.

Confirmed via mocked payloads only — no live RunPod deploy was run.

…N) scales to zero

Deploying with workers=(0, 1) omitted workersStandby from the saveEndpoint
payload, so the RunPod API applied its own default and the endpoint held a
warm worker forever instead of scaling to zero.

ServerlessResource now defaults workersStandby to workersMin and validates
explicit values against the (workersMin, workersMax) range, so the deployed
config matches the requested one. Endpoint also accepts an optional
workers_standby parameter for overriding the standby count explicitly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Endpoint(workers=(0, N)) deploys workersStandby: 1, so an app documented as scale-to-zero always holds a worker warm

1 participant