feat(deploy): provision client-mode Endpoint(image=...) resources during flash deploy - #370
Open
justinwlin wants to merge 4 commits into
Open
feat(deploy): provision client-mode Endpoint(image=...) resources during flash deploy#370justinwlin wants to merge 4 commits into
justinwlin wants to merge 4 commits into
Conversation
…g deployment An Endpoint declared in client mode (image=...) with no decorated functions produces a manifest with an empty resources map. flash deploy still uploaded the artifact, created app+environment, and printed 'deployed to production' even though nothing was provisioned. Detect the empty-resources case in _resolve_and_deploy after manifest validation and exit with an honest message: client-mode endpoints provision lazily on first run().
Endpoint(image=...) objects with no decorated functions produced a manifest with an empty resources map, so flash deploy uploaded and 'deployed' nothing. The build now discovers top-level client-mode Endpoint instances during scanning and registers each as a function-less manifest entry marked client_mode: true, so the existing deploy provisioning path (create_resource_from_manifest + ResourceManager.get_or_deploy_resource) provisions them like any other resource. Handler generators and local preview skip client_mode entries; there is no worker code to generate for them. - scanner: collect Endpoint(image=...) instances (id= excluded); tolerate same-name re-exports across files, fail on conflicting images - manifest: register client endpoints with deployment config extracted from the Endpoint's resource config; fall back to resource_type 'Endpoint' (resolved via gpuIds) when the built config class is not one the provisioner can rebuild - deployment: skip source-fingerprint injection for client_mode entries so code-only deploys do not churn external-service endpoints - deploy: keep the honest 'no deployable resources' message only for projects with neither decorated resources nor client endpoints Fixes #365
test_deploy_all_background spawned an untracked daemon thread that could
lose the scheduling race and run the real ResourceManager after the mock
patch and fixtures unwound. Its MagicMock(spec=ServerlessResource)
resources were then cached for cleanup, and _save_resources failed to
cloudpickle spec'd mocks ("args[0] from __newobj__ args has the wrong
class"), truncating the shared state file and breaking unrelated tests
downstream (REG008 PicklingError / "Ran out of input").
deploy_all_background now returns the spawned thread so callers can
await completion, and the test joins it while the get_or_deploy_resource
patch is still active, making the cross-test pollution structurally
impossible rather than a scheduling lottery.
justinwlin
marked this pull request as ready for review
August 25, 2026 19:09
- manifest: extract _derive_path_fields; the file-path/url-prefix/module-path derivation (with outside-project-root and relative-path fallbacks) was duplicated between _add_client_endpoints and build() - scanner: flatten _collect_client_endpoints dedup to a sequential guard chain (no behavior change) and drop the dead `or ""` on member.name — Endpoint.__init__ guarantees a name when image= is set - scanner tests: pin the two explicitly-coded branches — a same-file alias of one object records once; two same-name endpoints in the same file resolve last-wins (unlike the cross-file raise) - handler_generator: read client_mode via the hasattr-first dual-shape pattern used by the adjacent is_load_balanced check - deploy: capitalize the empty-resources short-circuit message to match neighboring user-facing prints
KAJdev
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #365 · Internal: CON-1230
An
Endpointdeclared in client mode (image=..., no decorated functions) produced a build manifest with an emptyresourcesmap.flash deployuploaded the artifact, created the app and environment, printed✓ deployed to production, and exited 0 — provisioning nothing.flash env get productionthen showedno resources. run flash deploy --env production, pointing the user back at the command that had just run.Fix
The build now discovers top-level client-mode
Endpoint(image=...)objects during scanning and registers each as a function-less manifest entry markedclient_mode: true. Deployment config —imageName,gpuIds/instanceIds, workers, scaler, env, volumes, datacenter locations — comes from the Endpoint's own_build_resource_config(), the same object first-run()provisioning uses.From there the existing deploy path (
reconcile_and_provision_resources→create_resource_from_manifest→ResourceManager.get_or_deploy_resource) provisions them like any other resource and persists endpoint URLs into the state manifest. Soflash env getlists them, and re-deploys reuse the same endpoint unless config changed.The first commit on this branch (f36fa48) shipped an honest "no deployable resources" message instead. That short-circuit remains, but now only fires for projects declaring neither decorated resources nor client-mode endpoints.
Verified
pytest tests/unit/cli --no-covpytest tests/unit --no-covruff check+ruff format, touched filesThe 3 failures are pre-existing
test_load_balancer_sls_stub.py×2 andtest_regressions.pyREG008 reproduce identically with these changes stashed — order-dependent pollution on the base commit, not introduced here.Offline end-to-end chain
Real
RuntimeScanner→ManifestBuilder.build()→create_resource_from_manifest()over a client-only project (GPUvllm-openai+ CPUwhisper) yields provisionableLiveServerless/CpuLiveServerlessobjects carrying the user image, GPU/CPU config, workers, and env. Verified withFLASH_IS_LIVE_PROVISIONINGunset (concrete class names) and=false(normalized toEndpoint, still provisionable).What changed
build_utils/scanner.py—RuntimeScannercollects top-levelEndpointinstances withimage=set andidunset (those attach to existing endpoints) intoscanner.client_endpointsduring module import. Re-exports (from defs import epin a second file) are tolerated when the image matches; the same name with a different image fails the build with a "defined in multiple files" error.build_utils/manifest.py—ManifestBuilder._add_client_endpointsregisters each client endpoint intoresourceswithfunctions: [],client_mode: true, and config extracted by re-importing the source module and calling_build_resource_config()(same_extract_config_propertiespath as decorated resources).resource_typerecords the concrete class only when the deploy-time provisioner can rebuild it; otherwise it falls back to"Endpoint", whichcreate_resource_from_manifestresolves to GPU/CPU viagpuIds. Name collisions with decorated resources raise.cli/commands/deploy.py— the empty-resources message is reworded to "no deployable resources found; nothing was deployed" and only triggers when the manifest has no resources at all.Four supporting changes
cli/utils/deployment.py—_FLASH_SOURCE_FINGERPRINTenv injection skipsclient_modeentries, so code-only deploys don't trigger pointless rolling updates of external-service endpoints. Reconciliation is otherwise unchanged, which is what keeps re-deploys idempotent (state-manifest compare +get_or_deploy_resourcereuse).build_utils/handler_generator.py— skipsclient_moderesources; no functions means no handler to generate.cli/commands/preview.py— local preview skipsclient_moderesources; they run user-supplied images and are provisioned remotely.runtime/resource_provisioner.py— supported class list is now a module-levelPROVISIONABLE_RESOURCE_TYPES. Behavior unchanged.Live testing
2026-08-25, real Runpod account. Client-mode-only app:
flash deploycreated appcon370-live+productionenv and provisioned and printed the endpoint:con370-live-1248 https://api.runpod.ai/v2/4kuwiutqisjt2f/runsync. Onmainthe same app yields emptyresourcesand a false success.GET /v1/endpoints/4kuwiutqisjt2f→workersMin: 0,workersMax: 1,idleTimeout: 60,gpuTypeIds: [A4000, A4500, 4000 Ada, 2000 Ada](the AMPERE_16 pool), flashboot on. Deployed config matches the request.flash env get productionlists the endpoint. Onmainit printedno resources.Out of scope
workersStandby: 1on the live endpoint is expected here and addressed in fix(serverless): default workersStandby to workersMin so workers=(0, N) scales to zero #372.