macOS: stale PID lock misjudged as alive after reboot due to PID reuse — server crash-loops and never self-heals

[e3322-pcsk9]

Summary

On macOS, after an unclean shutdown (reboot/power loss), the data-directory PID lock (.openviking.pid) is left behind. If the stale PID is reused by an unrelated process after reboot, _is_pid_alive() returns True for it, acquire_data_dir_lock() raises DataDirectoryLocked on every startup, and the server never recovers — under a service manager (launchd KeepAlive) this becomes an infinite crash loop.

The PID-reuse guard in _is_pid_alive() (verifying the process identity via /proc/<pid>/cmdline, see #1088) is Linux-only. macOS has no /proc, so on Darwin the check degrades to a bare os.kill(pid, 0) liveness probe — any live process holding the recycled PID (e.g. Spotlight's mdwrite) is misjudged as a running OpenViking instance.

Windows received the same class of fixes in #790 / #854 / #1201. macOS is the remaining uncovered platform.

Environment

• macOS (Apple Silicon, arm64)
• openviking 0.4.9.dev17 (editable install), still present on current main (6e944cc3, 2026-08-22)
• Service managed by launchd LaunchAgent (RunAtLoad=true, KeepAlive.SuccessfulExit=false, ThrottleInterval=10)

Real-world incident (what happened to me)

1. Server (PID 857) wrote .openviking.pid containing 857.
2. Machine rebooted; the process was killed before atexit/SIGTERM cleanup ran → stale lock file survived.
3. After reboot, macOS assigned PID 857 to mdwrite (Spotlight metadata writer, a long-lived system process).
4. Every subsequent launchd start: FastAPI lifespan → OpenVikingService() → acquire_data_dir_lock() → DataDirectoryLocked("Another OpenViking process (PID 857) ...") → uvicorn Application startup failed. Exiting. → exit code 3.
5. launchd retried every 10 s for ~7 days ≈ 50,000 crashes, producing ~330 MB of logs, with zero chance of self-healing (the lock holder mdwrite never exits).

Diagnosis was harder than necessary because the server prints OpenViking HTTP Server is running on 127.0.0.1:1933 to stdout before uvicorn.run(), while the actual traceback only lands in the file logger — the launchd-managed stdout log contained nothing but 50,000 copies of the banner.

Root cause

openviking/utils/process_lock.py:

def _is_pid_alive(pid: int) -> bool:
    ...
    try:
        os.kill(pid, 0)
    except ProcessLookupError:
        return False
    ...
    # PID exists, but on Linux PIDs are recycled. Verify this is actually
    # an OpenViking process by checking /proc/{pid}/cmdline ...
    if sys.platform.startswith("linux"):   # ← Darwin never enters here
        ...
    return True

Suggested fixes (either)

Option A — Darwin process-identity check, mirroring the Linux branch:

if sys.platform == "darwin":
    try:
        out = subprocess.run(
            ["sysctl", "-n", f"kern.proc.pid.{pid}"],
            capture_output=True, text=True, timeout=2,
        )
        name = (out.stdout or "").lower()
        if "openviking" not in name and "python" not in name:
            return False  # recycled PID held by an unrelated process
    except (OSError, subprocess.SubprocessError):
        pass

Option B — start-time nonce (stronger, cross-platform): record the holder process's start time (e.g. kern.proc.pid p_starttime on Darwin, /proc/<pid>/stat field 22 on Linux) in the lock file alongside the PID, and treat the lock as stale when the live process's start time differs. This closes the PID-reuse window entirely instead of relying on name matching.

Reproduction

1. Start openviking-server; note the PID written to <workspace>/.openviking.pid.
2. Kill it with kill -9 (or reboot) so cleanup never runs.
3. sudo -less trick to simulate reuse: keep starting/killing short-lived processes until one lands on the recorded PID — or simply reboot and wait for the OS to recycle it (on macOS, low PIDs are quickly reused by system daemons).
4. Start the server again → DataDirectoryLocked every time, until the innocent PID holder exits.

Happy to open a PR for Option A if the approach sounds right.