Skip to content

Overhaul provider streaming on a shared SSE base - #52

Merged
iskakaushik merged 1 commit into
mainfrom
provider-refresh/04-streaming
Aug 19, 2026
Merged

Overhaul provider streaming on a shared SSE base#52
iskakaushik merged 1 commit into
mainfrom
provider-refresh/04-streaming

Conversation

@iskakaushik

Copy link
Copy Markdown
Collaborator

The OpenAI and Anthropic stream implementations had drifted into
near-identical copies of the same threading, queueing, and SSE
parsing code, and both mishandled several streaming cases: tool-call
deltas were never assembled into events, finish events carried no
usage or finish reason, API error bodies on non-200 responses were
lost, and a stream could emit several terminal events.

Introduce providers::streaming::HttpSseStream, which owns the stream
thread, event queue, HTTP request and SSE line splitting, pending
tool-call assembly, and emit-once finish semantics. Each provider
implementation now only translates its SSE lines into stream events.

Behavioral fixes that come with the rework:

  • Streamed tool calls are accumulated per index and emitted as
    tool-call events, including when a gateway closes the stream
    without [DONE] or content_block_stop.
  • Exactly one finish event terminates every stream, carrying usage
    and the real finish reason on success and kFinishReasonError with
    no usage on failure. OpenAI streams now request
    stream_options.include_usage, without which the API omits usage
    from streaming responses entirely.
  • Error bodies of non-200 responses are captured via a response
    handler (httplib routes them through the content receiver, leaving
    response.body empty) so auth and rate-limit failures stay
    diagnosable. A user-initiated stop is no longer reported as a
    network error.
  • Anthropic SSE parsing tolerates present-but-null JSON fields from
    gateways, keeps a tool_use block's initial input separate from
    streamed input_json_delta fragments, completes on message_stop
    instead of waiting for the connection to close, and no longer
    races or terminates on a second start_stream call.

Stack created with GitHub Stacks CLIGive Feedback 💬

@iskakaushik
iskakaushik force-pushed the provider-refresh/04-streaming branch from be21385 to d11ed65 Compare August 19, 2026 17:43
@iskakaushik
iskakaushik force-pushed the provider-refresh/04-streaming branch from d11ed65 to 55ee2c5 Compare August 19, 2026 17:44
Base automatically changed from provider-refresh/03-model-catalog to main August 19, 2026 17:45
The OpenAI and Anthropic stream implementations had drifted into
near-identical copies of the same threading, queueing, and SSE
parsing code, and both mishandled several streaming cases: tool-call
deltas were never assembled into events, finish events carried no
usage or finish reason, API error bodies on non-200 responses were
lost, and a stream could emit several terminal events.

Introduce providers::streaming::HttpSseStream, which owns the stream
thread, event queue, HTTP request and SSE line splitting, pending
tool-call assembly, and emit-once finish semantics. Each provider
implementation now only translates its SSE lines into stream events.

Behavioral fixes that come with the rework:

- Streamed tool calls are accumulated per index and emitted as
  tool-call events, including when a gateway closes the stream
  without [DONE] or content_block_stop.
- Exactly one finish event terminates every stream, carrying usage
  and the real finish reason on success and kFinishReasonError with
  no usage on failure. OpenAI streams now request
  stream_options.include_usage, without which the API omits usage
  from streaming responses entirely.
- Error bodies of non-200 responses are captured via a response
  handler (httplib routes them through the content receiver, leaving
  response.body empty) so auth and rate-limit failures stay
  diagnosable. A user-initiated stop is no longer reported as a
  network error.
- Anthropic SSE parsing tolerates present-but-null JSON fields from
  gateways, keeps a tool_use block's initial input separate from
  streamed input_json_delta fragments, completes on message_stop
  instead of waiting for the connection to close, and no longer
  races or terminates on a second start_stream call.
@iskakaushik
iskakaushik force-pushed the provider-refresh/04-streaming branch from 55ee2c5 to 3260674 Compare August 19, 2026 17:45
@iskakaushik
iskakaushik merged commit 44fc774 into main Aug 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant