fix(sse): bound workspace SSE connection lifetime - #7058
Conversation
Teardown ran only from the request abort listener and the stream cancel callback, both of which fire only when the runtime reports a client disconnect. Nothing else bounded the connection, so a missed report left the pub/sub handler, the heartbeat timer, and the stream's undrained queue held for the life of the process. Add a jittered lifetime ceiling checked on the existing heartbeat tick, tighten reclaim for a vanished consumer via desiredSize, remove the abort listener on every teardown path, and run full teardown when a heartbeat enqueue fails. Log the close reason so opens minus closes is observable.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview Because forced reconnects can miss transient New Reviewed by Cursor Bugbot for commit cbe8898. Configure here. |
Greptile SummaryThis PR bounds workspace SSE connection retention and reconciles workspace chat lists after reconnect gaps.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| apps/sim/lib/events/sse-endpoint.ts | Adds bounded connection lifetime, unread-consumer detection, and centralized idempotent resource cleanup. |
| apps/sim/lib/events/sse-endpoint.test.ts | Covers expiry, unread detection, healthy draining, abort, cancellation, and overlapping teardown paths. |
| apps/sim/hooks/use-mothership-chat-events.ts | Adds reconnect-aware workspace-list invalidation while deliberately leaving chat-detail reconciliation unchanged. |
| apps/sim/hooks/use-mothership-chat-events.test.ts | Verifies that reconnect resynchronization invalidates workspace lists without touching chat details. |
Sequence Diagram
sequenceDiagram
participant Client as EventSource client
participant SSE as Workspace SSE
participant PubSub as Pub/Sub
participant Cache as Query cache
Client->>SSE: Open connection
PubSub-->>SSE: Transient workspace event
SSE-->>Client: SSE event
alt Client stops draining
SSE-->>SSE: Queue exceeds unread threshold
SSE-->>Client: Close stream
else Connection reaches ceiling
SSE-->>SSE: Jittered deadline expires
SSE-->>Client: Close stream
end
SSE->>PubSub: Unsubscribe and clear resources
Client->>SSE: Automatically reconnect
SSE-->>Client: Connection opens
Client->>Cache: Invalidate workspace chat lists
Reviews (5): Last reviewed commit: "fix(sse): raise the ceiling and narrow r..." | Re-trigger Greptile
task_status events are transient and never replayed, so any window with no open connection can drop a create, rename, delete, or completion. The chat hook reconnected silently and reconciled nothing, leaving list and detail caches stale until an unrelated action refreshed them. Resync on reconnect, on the first open of a re-subscription, and on a first open that only succeeded after an error, matching the pattern useMcpToolsEvents already uses for the same gap.
|
@cursor review |
The resync invalidated every chat detail, including one whose stream this client is rendering optimistically. Refetching there replaces the local transcript with a server copy that does not yet hold the in-flight message, which is exactly what status events avoid via shouldSkipDetailInvalidationForStreamEvent. Filter the detail invalidation with the same isLocalOptimisticActiveStream check. Those chats reconcile when their own stream finishes.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 70541f4. Configure here.
Optimistic markers alone were the skip condition, but a finished turn can leave activeStreamId and its live-assistant message cached when finalization skips detail invalidation for a queued follow-up. That chat would then be excluded from every future resync — permanently, since only a refetch clears the markers, and the resync was the refetch. Gate the skip on a non-terminal streamSnapshot status so it covers turns that are genuinely still streaming. Exports isTerminalStreamStatus, which was already the private check for this in effective-transcript.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit f76765f. Configure here.
Deciding from cache whether a chat is still streaming is not reliable — the optimistic markers outlive the turn, and each refinement of that predicate exposed another state where it answers wrongly. Drop it: the resync now invalidates only the workspace lists, which is always safe, and chat detail reconciliation stays as it is today rather than being half-solved here. Raise the ceiling to 4h, matching lib/realtime/event-stream-route.ts. A healthy client is drained and so is never unread; the unread check is what reclaims a vanished consumer, and it does so within minutes. A short ceiling would therefore only force reconnects on the connections that are working, and every reconnect is a window where a transient event can be missed. Retention stays bounded by the ceiling instead of by process uptime.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit cbe8898. Configure here.
Summary
createWorkspaceSSEtore down a connection only from therequest.signalabort listener and the stream'scancel(). Both fire only when the runtime reports the client disconnect, and nothing else bounded the connection — so a missed report left the pub/sub handler in a process-lifetimeSet, the heartbeat interval running, and the stream's undrained queue held until the process restarted. Retention scaled with uptime instead of with concurrent clients.controller.desiredSize(the default strategy reports1 - queued) and closes once nothing has been pulled for several minutes. This is what actually reclaims a vanished consumer, within minutes. A healthy client keeps its stream drained and is never unread, so this never touches a working connection.lib/realtime/event-stream-route.ts, which uses the same deadline pattern for the same purpose. Deliberately far longer than the unread window: a short ceiling would only force reconnects on the connections that are working, and every reconnect is a window in which a transient event can be missed.abortlistener is now removed on every path.{ once: true }only self-removes if abort fires; the ceiling and unread paths close while the signal is still live, so the listener would have stayed installed retainingclose→ the controller, the teardown list, and the workspace id — most of the scope this change exists to release. It registers with an ownedAbortControlleras its removal token.enqueuenow runs full teardown instead of clearing only its own interval and leaving the subscriptions registered.cleanuptakes a reason, so the close log says which path ran (aborted,cancelled,expired,unread,errored). Opens minus closes, grouped by reason, is what makes this class of leak visible rather than inferred.useMothershipChatEvents) —task_statusevents are transient and never replayed, and the hook previously reconciled nothing on reconnect. It now re-syncs the workspace chat lists on reconnect, on the first open of a re-subscription, and on a first open that only succeeded after an error, matching the patternuseMcpToolsEventsalready uses. Only the lists: they carry the create/rename/delete state a gap can drop and are always safe to refetch.Chat detail reconciliation is deliberately out of scope and unchanged from
staging. Refetching a mounted detail can replace an in-flight optimistic transcript with a server copy that lacks the streaming message, and cached state cannot reliably say whether a turn is still running — the optimistic markers outlive it. Doing that properly needs the streaming state that can answer it, and belongs in its own change rather than being half-solved here.createWorkspaceSSEis the only push-driven, indefinite-lifetime SSE surface in the app. The two other long-lived SSE routes are poll loops that already carry their own duration ceilings and never depended on abort. A shared bounded-SSE primitive would be the honest end state for all three, but that is a three-site refactor and doesn't belong in a leak fix.Type of Change
Testing
sse-endpoint.test.tscovers the ceiling, the unread path, a drained connection surviving past the unread threshold, abort, consumer cancel, and teardown idempotency when two paths overlap.use-mothership-chat-events.test.tscovers the resync invalidating the lists and leaving details untouched.Verified the tests are load-bearing rather than assuming it: disabling each backstop reds exactly its own test and leaves the rest green. The abort-listener removal isn't unit-observable (
AbortSignalexposes no listener count), so it's covered by review — I dropped an earlier test that looked like it covered it but only re-asserted an idempotency guard another test already owns.bun run lint,bun run check:audits(33 audits), andtscclean. 163 test files / 1816 tests green acrosshooks,lib/events, andlib/copilot/chat.Checklist