fix: run each matrix leg on its own Python version, and stop a leaked test thread

[runpod-Henrik]

Two independent fixes, one commit each.

1. The Python version matrix was decorative

quality-gates declares ['3.10', '3.11', '3.12', '3.13'], but every leg ran Python 3.11. make dev runs a bare uv sync, and uv resolves its interpreter from .python-version (pinned to 3.11) rather than from the one actions/setup-python just installed.

Confirmed across four CI runs on three unrelated branches, the oldest from 2026-08-10 — every leg reports Using CPython 3.11.x and builds .venv/lib/python3.11:

Run	Branch	Leg that failed	Interpreter
31410374670	deanq/sls-python-parity-by-default	Quality Gates (3.10)	Python 3.11.15
32861648254	fix/367-app-delete-removes-endpoint	Quality Gates (3.12)	Python 3.11.16
32869479106	fix/365-deploy-empty-resources	Quality Gates (3.12)	Python 3.11.16
33012747552	Henrik/ci-coverage-artifact	Quality Gates (3.13)	Python 3.11.16

Fix: UV_PYTHON: ${{ matrix.python-version }}, which takes precedence over the pin file. Set at job level rather than on the install step, because make ci-quality-github runs uv run pytest, which resolves the interpreter again.

uv.lock was stale, and 3.13 could not resolve without it

Re-locked in the same commit. It was wrong in two ways:

• requires-python = ">=3.10, <3.13" while pyproject.toml says >=3.10,<3.14 — the lock never covered 3.13 at all
• tomlkit>=0.13.0 is declared in pyproject.toml but was missing from the runpod-flash dependency and requires-dist lists

uv lock --check   (previous lock)  -> exit 1, "needs to be updated"
uv lock --check   (this lock)      -> passes on 3.10, 3.11, 3.12, 3.13

CI never caught the drift: pre-check uses --frozen, which does not verify freshness, and make dev uses plain uv sync, which silently re-resolves on the runner.

2. test_deploy_all_background leaked a thread into other tests

This was the cause of a flaky failure that landed on an unrelated test, usually:

FAILED tests/unit/test_regressions.py::TestREG008NonInteractiveEnvDelete::test_undeploy_resource_force_remove_no_tty
_pickle.PicklingError: args[0] from __newobj__ args has the wrong class

deploy_all_background() starts a daemon thread and returns immediately, and the test never joined it. The patch.object(..., "get_or_deploy_resource") context was lifted while the thread was still starting, so the thread ran against the unpatched manager and registered the fixture's MagicMock(spec=ServerlessResource) objects into the ResourceManager singleton — during whichever unrelated test happened to be running at that moment.

ResourceManager._save_resources() cloudpickles its entire state on every registration, and a MagicMock cannot be pickled: obj.__class__ is the spec'd class while type(obj) is MagicMock, exactly the mismatch pickle.save_reduce rejects. The mock arrived as a dict key, via _migrate_to_name_based_keys() calling resource.get_resource_key() on it — hence the key rendering as <MagicMock name='mock.get_resource_key()'>.

Because it depended on thread scheduling and on xdist's dynamic worker assignment, the victim, the worker and the matrix leg all varied per run — see the table above (legs 3.10/3.12/3.13, workers gw0/gw2/gw3). Forcing everything into one process reproduced it every time; -n 4 reproduced it in 0 of 6 runs.

threading.Thread is now stubbed, which keeps what the test actually asserts — the call is non-blocking and spawns a daemon thread — and lets nothing escape. The test previously asserted nothing at all; its own comment said "not much we can test here without waiting for thread".

Verification

Check	Result
Full suite, single-process, 3.11 (the reproducer)	2680 passed, 0 PicklingError
Full suite, single-process, real Python 3.13	2680 passed, 0 PicklingError, no 3.13-specific failures
Serial pass (-m serial)	53 passed, 1 skipped
ruff format --check + ruff check, 260 tracked files	clean

The same single-process run reproduced the failure before the change, so this is a before/after comparison rather than an absence of evidence.

Not addressed here

Two deeper faults behind this class of bug, both wider changes than a flake fix:

• ResourceManager._resources / _resource_configs are class variables, so instance mutation writes through to global state — while _load_resources() uses assignment, which shadows them with instance attributes. The manager silently switches between class-level and instance-level state depending on whether the state file existed.
• conftest.worker_flash_dir is scope="session", so every test in a worker shares one resources.pkl, and reset_singletons forces a reload from it on every test.

🤖 Generated with Claude Code

[runpod-Henrik]

What the real matrix found on 3.10, and the fix

Making the matrix real (259c527) turned the 3.10 leg into an actual 3.10 run for the first time, and it failed immediately:

Using CPython 3.10.21
platform linux -- Python 3.10.21
= 3 failed, 2627 passed, 1 skipped, 1 xfailed in 127.20s

All three failures are pre-existing and 3.10-specific — nothing to do with the matrix change itself, which only stopped hiding them. All three reproduce on a real 3.10 interpreter independently of this branch. 0625cc9 fixes them; both are test-only changes.

1 & 2. test_run_server_helpers.py — inspect.signature() on a Mock

FAILED TestCallWithBodyEmptyInputValidation::test_allows_plain_dict_body
       AssertionError: assert <starlette.responses.JSONResponse object> == {'ok': True}
FAILED TestCallWithBodyEmptyInputValidation::test_allows_empty_plain_dict_body

call_with_body → _map_body_to_params calls inspect.signature(func), and signature() of a Mock is version-dependent:

Python	inspect.signature(AsyncMock())
3.10.19	raises TypeError: 'Mock' object is not subscriptable
3.13.11	returns (*args, **kwargs)

Mock auto-creates a __signature__ child attribute, which inspect then tries to use. Passing spec= does not help — verified; the failure is inside signature(), not in spec resolution.

This was double-masked. call_with_body catches Exception and converts it into a 500 JSONResponse, so the visible failure was the uninformative assert <JSONResponse object> == {'ok': True}. The real TypeError was readable only inside the response body:

returned: JSONResponse
body: {"error":"'Mock' object is not subscriptable"}
_map_body_to_params raised TypeError: 'Mock' object is not subscriptable   # py3.10
_map_body_to_params -> {'args': {'key': 'value'}}                         # py3.13

Fix: wrap the mock in a real async def (*args, **kwargs). That is the same signature 3.11+ inferred from the Mock, so both branches of _map_body_to_params behave exactly as before — the non-empty body maps to the first parameter, the empty body spreads as kwargs — and delegating to the mock keeps assert_called_once_with().

3. test_resource_manager_extended.py — pickling a Mock

FAILED TestLoadResources::test_loads_legacy_dict_format
       _pickle.PicklingError: Could not pickle object as excessively deep recursion required.

The test does cloudpickle.dump({"key1": MagicMock(config_hash="hash1")}). Whether cloudpickle can pickle a Mock is version-dependent; 3.10 rejects it.

Fix: a module-level _LegacyResource stand-in. Only .config_hash is read (via _refresh_config_hashes), and the absence of get_resource_key is what keeps the legacy key un-migrated — which is exactly what the test asserts. Module-level rather than defined in the test body, because cloudpickle serialises a function-local class by value.

Verification

CI on 0625cc9 — the first run in this repo's history where all four legs use four different interpreters:

Leg	Result
Quality Gates (3.10)	pass
Quality Gates (3.11)	pass
Quality Gates (3.12)	pass
Quality Gates (3.13)	pass
Validation	pass

Locally, before pushing, both affected files were run against three real interpreters:

Python	Result
3.10	40 passed (previously 3 failed)
3.11	40 passed
3.13	40 passed

ruff format --check and ruff check clean.

Worth a separate look

The blanket except Exception: return JSONResponse(status_code=500, ...) in call_with_body turns genuine errors into silent 500s. It is what made a one-line TypeError take three steps to diagnose, and in production it would do the same to a real bug. Not changed here — it is behaviour, not a test fix.