Skip to content

feat(deploy): provision client-mode Endpoint(image=...) resources during flash deploy - #370

Open
justinwlin wants to merge 4 commits into
mainfrom
fix/365-deploy-empty-resources
Open

feat(deploy): provision client-mode Endpoint(image=...) resources during flash deploy#370
justinwlin wants to merge 4 commits into
mainfrom
fix/365-deploy-empty-resources

Conversation

@justinwlin

@justinwlin justinwlin commented Aug 25, 2026

Copy link
Copy Markdown

Problem

Fixes #365 · Internal: CON-1230

An Endpoint declared in client mode (image=..., no decorated functions) produced a build manifest with an empty resources map. flash deploy uploaded the artifact, created the app and environment, printed ✓ deployed to production, and exited 0 — provisioning nothing. flash env get production then showed no resources. run flash deploy --env production, pointing the user back at the command that had just run.

Fix

The build now discovers top-level client-mode Endpoint(image=...) objects during scanning and registers each as a function-less manifest entry marked client_mode: true. Deployment config — imageName, gpuIds/instanceIds, workers, scaler, env, volumes, datacenter locations — comes from the Endpoint's own _build_resource_config(), the same object first-run() provisioning uses.

From there the existing deploy path (reconcile_and_provision_resourcescreate_resource_from_manifestResourceManager.get_or_deploy_resource) provisions them like any other resource and persists endpoint URLs into the state manifest. So flash env get lists them, and re-deploys reuse the same endpoint unless config changed.

The first commit on this branch (f36fa48) shipped an honest "no deployable resources" message instead. That short-circuit remains, but now only fires for projects declaring neither decorated resources nor client-mode endpoints.

Verified

Check Result
pytest tests/unit/cli --no-cov 587 passed
pytest tests/unit --no-cov 2575 passed, 3 failed (pre-existing)
ruff check + ruff format, touched files clean
Offline end-to-end chain provisionable objects produced
The 3 failures are pre-existing

test_load_balancer_sls_stub.py ×2 and test_regressions.py REG008 reproduce identically with these changes stashed — order-dependent pollution on the base commit, not introduced here.

Offline end-to-end chain

Real RuntimeScannerManifestBuilder.build()create_resource_from_manifest() over a client-only project (GPU vllm-openai + CPU whisper) yields provisionable LiveServerless/CpuLiveServerless objects carrying the user image, GPU/CPU config, workers, and env. Verified with FLASH_IS_LIVE_PROVISIONING unset (concrete class names) and =false (normalized to Endpoint, still provisionable).

What changed

build_utils/scanner.pyRuntimeScanner collects top-level Endpoint instances with image= set and id unset (those attach to existing endpoints) into scanner.client_endpoints during module import. Re-exports (from defs import ep in a second file) are tolerated when the image matches; the same name with a different image fails the build with a "defined in multiple files" error.

build_utils/manifest.pyManifestBuilder._add_client_endpoints registers each client endpoint into resources with functions: [], client_mode: true, and config extracted by re-importing the source module and calling _build_resource_config() (same _extract_config_properties path as decorated resources). resource_type records the concrete class only when the deploy-time provisioner can rebuild it; otherwise it falls back to "Endpoint", which create_resource_from_manifest resolves to GPU/CPU via gpuIds. Name collisions with decorated resources raise.

cli/commands/deploy.py — the empty-resources message is reworded to "no deployable resources found; nothing was deployed" and only triggers when the manifest has no resources at all.

Four supporting changes
  • cli/utils/deployment.py_FLASH_SOURCE_FINGERPRINT env injection skips client_mode entries, so code-only deploys don't trigger pointless rolling updates of external-service endpoints. Reconciliation is otherwise unchanged, which is what keeps re-deploys idempotent (state-manifest compare + get_or_deploy_resource reuse).
  • build_utils/handler_generator.py — skips client_mode resources; no functions means no handler to generate.
  • cli/commands/preview.py — local preview skips client_mode resources; they run user-supplied images and are provisioned remotely.
  • runtime/resource_provisioner.py — supported class list is now a module-level PROVISIONABLE_RESOURCE_TYPES. Behavior unchanged.

Live testing

2026-08-25, real Runpod account. Client-mode-only app:

Endpoint(name="con370-live-1248", image="runpod/serverless-hello-world:0.4.1",
         gpu=GpuGroup.AMPERE_16, workers=(0, 1), idle_timeout=60)
  • flash deploy created app con370-live + production env and provisioned and printed the endpoint: con370-live-1248 https://api.runpod.ai/v2/4kuwiutqisjt2f/runsync. On main the same app yields empty resources and a false success.
  • GET /v1/endpoints/4kuwiutqisjt2fworkersMin: 0, workersMax: 1, idleTimeout: 60, gpuTypeIds: [A4000, A4500, 4000 Ada, 2000 Ada] (the AMPERE_16 pool), flashboot on. Deployed config matches the request.
  • flash env get production lists the endpoint. On main it printed no resources.

Out of scope

…g deployment

An Endpoint declared in client mode (image=...) with no decorated
functions produces a manifest with an empty resources map. flash deploy
still uploaded the artifact, created app+environment, and printed
'deployed to production' even though nothing was provisioned.

Detect the empty-resources case in _resolve_and_deploy after manifest
validation and exit with an honest message: client-mode endpoints
provision lazily on first run().
Endpoint(image=...) objects with no decorated functions produced a
manifest with an empty resources map, so flash deploy uploaded and
'deployed' nothing. The build now discovers top-level client-mode
Endpoint instances during scanning and registers each as a
function-less manifest entry marked client_mode: true, so the
existing deploy provisioning path (create_resource_from_manifest +
ResourceManager.get_or_deploy_resource) provisions them like any
other resource. Handler generators and local preview skip
client_mode entries; there is no worker code to generate for them.

- scanner: collect Endpoint(image=...) instances (id= excluded);
  tolerate same-name re-exports across files, fail on conflicting
  images
- manifest: register client endpoints with deployment config
  extracted from the Endpoint's resource config; fall back to
  resource_type 'Endpoint' (resolved via gpuIds) when the built
  config class is not one the provisioner can rebuild
- deployment: skip source-fingerprint injection for client_mode
  entries so code-only deploys do not churn external-service
  endpoints
- deploy: keep the honest 'no deployable resources' message only for
  projects with neither decorated resources nor client endpoints

Fixes #365
@justinwlin justinwlin changed the title fix(deploy): report client-mode endpoints honestly instead of claiming deployment feat(deploy): provision client-mode Endpoint(image=...) resources during flash deploy Aug 25, 2026
@justinwlin
justinwlin requested a review from deanq August 25, 2026 17:08
test_deploy_all_background spawned an untracked daemon thread that could
lose the scheduling race and run the real ResourceManager after the mock
patch and fixtures unwound. Its MagicMock(spec=ServerlessResource)
resources were then cached for cleanup, and _save_resources failed to
cloudpickle spec'd mocks ("args[0] from __newobj__ args has the wrong
class"), truncating the shared state file and breaking unrelated tests
downstream (REG008 PicklingError / "Ran out of input").

deploy_all_background now returns the spawned thread so callers can
await completion, and the test joins it while the get_or_deploy_resource
patch is still active, making the cross-test pollution structurally
impossible rather than a scheduling lottery.
@justinwlin
justinwlin marked this pull request as ready for review August 25, 2026 19:09
- manifest: extract _derive_path_fields; the file-path/url-prefix/module-path
  derivation (with outside-project-root and relative-path fallbacks) was
  duplicated between _add_client_endpoints and build()
- scanner: flatten _collect_client_endpoints dedup to a sequential guard
  chain (no behavior change) and drop the dead `or ""` on member.name —
  Endpoint.__init__ guarantees a name when image= is set
- scanner tests: pin the two explicitly-coded branches — a same-file alias
  of one object records once; two same-name endpoints in the same file
  resolve last-wins (unlike the cross-file raise)
- handler_generator: read client_mode via the hasattr-first dual-shape
  pattern used by the adjacent is_load_balanced check
- deploy: capitalize the empty-resources short-circuit message to match
  neighboring user-facing prints
@justinwlin
justinwlin requested a review from KAJdev August 25, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flash deploy reports success for a client-mode (image=) app but provisions no endpoint

2 participants