A recoverable FFI timeout is reported as `Panic`, and the SIGTERM that follows is only safe when the SDK owns a disposable process

[mtc-markvidal]

Summary

Two related things, both narrower than they first look:

1. livekit-ffi reports the room-event-ready timeout through send_panic, on a code
   path where nothing panicked and everything has already been cleaned up.
2. ffi_event_callback answers every Panic with os.kill(os.getpid(), SIGTERM).
   That is coherent when the SDK runs in a disposable per-job subprocess — but
   livekit-agents does not always run it there. Under JobExecutorType.THREAD all jobs
   share the agent server process, so one session's stranded handshake signals the
   whole worker.

I am not asking you to abandon "panic means exit". For watch_panic it looks right, and
watch_panic says so itself. I am asking that a recoverable timeout not use that
channel, because the SIGTERM policy downstream of it cannot distinguish the two.

Environment: livekit 1.1.14, CPython 3.14, livekit/livekit-server v1.13.5
--dev, Linux.

1. The room-event-ready timeout is not a panic

livekit-ffi/src/server/room.rs:248-266:

if tokio::time::timeout(
    ROOM_EVENT_READY_TIMEOUT,
    ffi_room.room_event_ready_notify.notified(),
).await.is_err() {
    // ...
    drop(handle);
    ffi_room.close(server, DisconnectReason::ConnectionTimeout).await;
    server.drop_handle(handle_id);
    server.send_panic(Box::new(FfiError::InvalidRequest(msg.into())));
    return;
}

close → drop_handle → then report → return. No Rust panic occurs anywhere on
this path; no lock is poisoned, no task is aborted, no unwind is caught. The event is a
diagnostic wearing the word panic because the FFI has no other channel for reporting an
internal failure.

The three producers of Panic are not equivalent, and I think that is the whole issue:

producer	what happened	is "exit the process" right?
server/mod.rs:286-301 watch_panic	a tokio task really panicked; re-panic!s after reporting	yes — and the code says so: "Recommended behaviour is to exit the process"
cabi.rs:63-104	catch_unwind caught an unwind mid-request; returns INVALID_HANDLE and continues	arguable — contained, but request state may be inconsistent
server/room.rs:248-266	a tokio::time::timeout elapsed on a fully-closed, fully-unwound path	no — nothing is degraded

Measured: the process is fine afterwards

Strand the handshake (see #784 — cancel connect, wait out the 15s window), catch
the SIGTERM instead of dying of it, then keep using the SDK on the same FfiClient:

[  0.01s] connect cancelled -- handshake now stranded
livekit_ffi::server::room:261 - timed out waiting for ReadyForRoomEventRequest after ConnectCallback (room_handle=3)
FFI Panic:  invalid request: timed out waiting for ReadyForRoomEventRequest after ConnectCallback (room_handle=3)
[ 15.09s] SIGTERM from the SDK -- declining it
[ 15.09s] using the SDK after the 'unrecoverable' panic
[ 15.12s] connected: sid=RM_AGTWUdJm5QpV identity=after-panic
[ 15.16s] published track sid=TR_VCZ6LT9AUV5yx8
[ 16.16s] disconnected cleanly -- the FFI was never in an unrecoverable state
exit 0

A fresh Room.connect succeeds, room.sid resolves, a video track publishes, and
disconnect() is clean. Exit 0.

repro_panic.py

import asyncio, signal, time
from livekit import api, rtc

URL = "ws://127.0.0.1:7880"
t0 = time.monotonic()

def log(msg):
    print(f"[{time.monotonic() - t0:6.2f}s] {msg}", flush=True)

def token(identity, room):
    return (api.AccessToken("devkey", "secret")
            .with_identity(identity)
            .with_grants(api.VideoGrants(room_join=True, room=room))
            .to_jwt())

async def main():
    got_sigterm = asyncio.Event()
    asyncio.get_running_loop().add_signal_handler(
        signal.SIGTERM,
        lambda: (log("SIGTERM from the SDK -- declining it"), got_sigterm.set()),
    )

    # Strand the handshake: one tick in, cancel. See the linked issue.
    room = rtc.Room()
    join = asyncio.create_task(room.connect(URL, token("stranded", "repro-room")))
    await asyncio.sleep(0)
    join.cancel()
    try:
        await join
    except asyncio.CancelledError:
        log("connect cancelled -- handshake now stranded")

    await asyncio.wait_for(got_sigterm.wait(), timeout=30)

    log("using the SDK after the 'unrecoverable' panic")
    healthy = rtc.Room()
    await healthy.connect(URL, token("after-panic", "after-panic-room"))
    log(f"connected: sid={await healthy.sid} identity={healthy.local_participant.identity}")

    source = rtc.VideoSource(320, 240)
    pub = await healthy.local_participant.publish_track(
        rtc.LocalVideoTrack.create_video_track("v", source),
        rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_CAMERA),
    )
    log(f"published track sid={pub.sid}")

    await asyncio.sleep(1)  # let the publisher finish negotiating
    await healthy.disconnect()
    log("disconnected cleanly -- the FFI was never in an unrecoverable state")

asyncio.run(main())

Run with uv run --with livekit --with livekit-api python repro_panic.py; echo "exit $?"
against docker run --rm -p 7880:7880 livekit/livekit-server:v1.13.5 --dev --bind 0.0.0.0.

2. The SIGTERM is only safe when the process is disposable

# livekit/rtc/_ffi_client.py:188-192
elif which == "panic":
    print("FFI Panic: ", event.panic.message, file=sys.stderr, flush=True)
    # We are in a unrecoverable state, terminate the process
    os.kill(os.getpid(), signal.SIGTERM)
    return

Under JobExecutorType.PROCESS this is defensible: the job subprocess is isolated, and
per the deployment docs a job crash "doesn't affect the agent server or other jobs".

But livekit-agents also runs jobs as threads in the agent server process:

• livekit-agents/livekit/agents/worker.py — JobExecutorType.THREAD is the
  default on Windows (if sys.platform.startswith("win")).
• cli/cli.py and cli/_legacy.py set it for console mode.
• jupyter.py sets it for notebook use.

In that mode os.getpid() is the agent server, and the framework's own reading of SIGTERM
is "agent servers stop accepting jobs upon SIGINT or SIGTERM". So one stranded
handshake in one job silently retires an entire worker: it stays up, finishes what it has,
and quietly stops taking work. I have not measured this path — the reasoning is from
worker.py, proc_pool.py and your deployment docs, so please sanity-check it — but if
it holds, the policy in _ffi_client.py is making an assumption the framework does not
always satisfy.

What would help

1. Don't route the room-event-ready timeout through send_panic. An error on the connect,
   a room event, or any distinct severity the client can tell apart from a real panic
   would let the SDK stop treating it as fatal.
2. If (1) lands, _ffi_client.py needs no change at all — which is why I think (1) is the
   real fix rather than adding a policy knob.
3. If the panic channel stays, consider whether ffi_event_callback's SIGTERM should be
   conditional on the executor type, or at least documented alongside
   JobExecutorType.THREAD.

Related

#784 is the bug that strands the handshake in the first place, without any misuse of
the room API — and it reaches JobContext.connect() in livekit-agents, which awaits
room.connect unguarded. The two are independent: fixing either one alone removes the
process death, but only fixing #784 leaves the mislabelled severity in place for
every other way the handshake could be missed.