Skip to content

fix(model-router): preserve upstream response headers - #2992

Open
tang-vu wants to merge 2 commits into
Osmantic:mainfrom
tang-vu:fix/model-router-preserve-response-headers
Open

fix(model-router): preserve upstream response headers#2992
tang-vu wants to merge 2 commits into
Osmantic:mainfrom
tang-vu:fix/model-router-preserve-response-headers

Conversation

@tang-vu

@tang-vu tang-vu commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Why this matters

The model router preserved upstream status codes and bodies but discarded end-to-end response metadata. In particular, a runtime returning 429/503 lost Retry-After, request correlation IDs, and cache directives, leaving OpenAI-compatible clients unable to back off correctly or correlate failures.

Root cause and invariant

Both streaming and buffered response constructors passed only ODS-generated headers. The invariant is that end-to-end upstream headers survive both response paths, while hop-by-hop fields, connection-nominated fields, stale body framing/encoding, and ODS-owned metadata never cross incorrectly.

A shared response filter now removes hop-by-hop headers, Content-Length, Content-Encoding, every token named by Connection, and case-insensitive collisions with ODS headers. Remaining headers are merged into both response paths.

Overlap check

Searched open and closed PRs for model router response headers, Retry-After, and the changed production file. Same-file open PRs are #2843 (MLX runtime family) and #2719 (internal-key auth); neither covers response metadata. Token Spy header PR #2971 affects a different proxy and implementation.

Regression coverage

A FastAPI boundary test covers buffered and streaming requests. It asserts preservation of Retry-After, an upstream request ID, and Cache-Control, while proving Connection, its nominated private header, and a stale Content-Length do not leak. The full router suite also protects existing ODS header behavior; an initial duplicate-case collision was caught and fixed before commit.

Validation

  • pytest -q ods/extensions/services/model-router/tests/test_router.py -x — 50 passed
  • python -m py_compile ods/extensions/services/model-router/app/main.py ods/extensions/services/model-router/tests/test_router.py
  • git diff --check

Tradeoffs and rollback

The filter preserves arbitrary end-to-end extension headers because model runtimes use provider-specific rate-limit and correlation names. Body framing is always regenerated after decoding/rewriting. Revert is isolated and requires no state migration.

@tang-vu

tang-vu commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Batch compatibility receipt (2026-08-23)

Validated merge order: #2989 ? #2990 ? #2993 ? #2991 ? #2992 ? #2994 ? #2995 ? #2996 ? #2997 ? #2998. The changes are independently useful; this order only reconciles shared model-router and magic-link files.

Synthetic integration head: origin/batch/quality-ten-20260822-round2-integration at 91eb730d. The only textual conflict was the two model-router tests inserting at the same class boundary; the integration resolution retains both contracts. Magic-link changes merged cleanly.

Combined validation on that exact head:

  • remote-provider egress contract: 16 passed
  • Token Spy suites: 22 passed, 1 skipped (live PostgreSQL availability)
  • model-router suite: 55 passed
  • APE suite: 29 passed
  • dashboard OAuth + magic-link suites: 118 passed, 2 platform skips
  • Brave Search loopback E2E: all checks passed
  • git diff --check: passed

All required GitHub checks are green across the batch. #2992 initially hit a transient openSUSE repository/mirror failure while installing rsync; a clearly labeled empty retry commit reran the unchanged tree, and openSUSE plus the full matrix passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant