Overhaul provider streaming on a shared SSE base - #52
Merged
Conversation
iskakaushik
force-pushed
the
provider-refresh/04-streaming
branch
from
August 19, 2026 17:43
be21385 to
d11ed65
Compare
iskakaushik
force-pushed
the
provider-refresh/04-streaming
branch
from
August 19, 2026 17:44
d11ed65 to
55ee2c5
Compare
The OpenAI and Anthropic stream implementations had drifted into near-identical copies of the same threading, queueing, and SSE parsing code, and both mishandled several streaming cases: tool-call deltas were never assembled into events, finish events carried no usage or finish reason, API error bodies on non-200 responses were lost, and a stream could emit several terminal events. Introduce providers::streaming::HttpSseStream, which owns the stream thread, event queue, HTTP request and SSE line splitting, pending tool-call assembly, and emit-once finish semantics. Each provider implementation now only translates its SSE lines into stream events. Behavioral fixes that come with the rework: - Streamed tool calls are accumulated per index and emitted as tool-call events, including when a gateway closes the stream without [DONE] or content_block_stop. - Exactly one finish event terminates every stream, carrying usage and the real finish reason on success and kFinishReasonError with no usage on failure. OpenAI streams now request stream_options.include_usage, without which the API omits usage from streaming responses entirely. - Error bodies of non-200 responses are captured via a response handler (httplib routes them through the content receiver, leaving response.body empty) so auth and rate-limit failures stay diagnosable. A user-initiated stop is no longer reported as a network error. - Anthropic SSE parsing tolerates present-but-null JSON fields from gateways, keeps a tool_use block's initial input separate from streamed input_json_delta fragments, completes on message_stop instead of waiting for the connection to close, and no longer races or terminates on a second start_stream call.
iskakaushik
force-pushed
the
provider-refresh/04-streaming
branch
from
August 19, 2026 17:45
55ee2c5 to
3260674
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OpenAI and Anthropic stream implementations had drifted into
near-identical copies of the same threading, queueing, and SSE
parsing code, and both mishandled several streaming cases: tool-call
deltas were never assembled into events, finish events carried no
usage or finish reason, API error bodies on non-200 responses were
lost, and a stream could emit several terminal events.
Introduce providers::streaming::HttpSseStream, which owns the stream
thread, event queue, HTTP request and SSE line splitting, pending
tool-call assembly, and emit-once finish semantics. Each provider
implementation now only translates its SSE lines into stream events.
Behavioral fixes that come with the rework:
tool-call events, including when a gateway closes the stream
without [DONE] or content_block_stop.
and the real finish reason on success and kFinishReasonError with
no usage on failure. OpenAI streams now request
stream_options.include_usage, without which the API omits usage
from streaming responses entirely.
handler (httplib routes them through the content receiver, leaving
response.body empty) so auth and rate-limit failures stay
diagnosable. A user-initiated stop is no longer reported as a
network error.
gateways, keeps a tool_use block's initial input separate from
streamed input_json_delta fragments, completes on message_stop
instead of waiting for the connection to close, and no longer
races or terminates on a second start_stream call.
Stack created with GitHub Stacks CLI • Give Feedback 💬