fix(#4060): extend backoff gate to remote MCP HTTP errors

[aheritier]

🤖 Automated implementer agent — this comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Stacked on PR #4062. Base retargets to main once that merges.

Refs #4060 (partial — A2A pacing and LSP crash-loop pacing deferred; see below)

What

Remote MCP servers responding with 503/429/5xx during the initialize handshake previously triggered a fresh connect attempt on every agent turn (the #4060 burst pattern). This PR fixes that by wrapping the HTTP status from the remote server in a *modelerrors.StatusError so the StartableToolSet backoff gate arming logic can pace retries — including a server-supplied Retry-After hint, and correctly for responses with no body.

Design

Wrap point — enrichConnectError in pkg/tools/mcp/remote.go. The oauthTransport already records the last HTTP error status via logErrorResponse (for any >= 400 response). enrichConnectError wraps that status (regardless of whether the response carried a body — many rate-limit/load-balancer responses don't) via modelerrors.WrapHTTPError, surfacing it as a *StatusError in the chain.

Retry-After is honored. pkg/tools/mcp/oauth.go's oauthTransport now also captures the raw Retry-After header value alongside the status/body it already tracked. lastServerErrorSnapshot() reads status, message, and Retry-After together under a single lock (not three separate accessor calls) so a caller can never pair a status from one response with a Retry-After header captured from a different concurrent response on the same transport — this transport's RoundTrip can run concurrently for one logical connect attempt (e.g. a standalone SSE probe racing the initialize call). enrichConnectError builds a minimal *http.Response carrying that header and passes it to WrapHTTPError, matching the handling already in place for model-provider adapters (PR #4062).

What arms the gate. startBackoffRetryable checks for a *modelerrors.StatusError with a retryable HTTP status (429/408/5xx) via errors.As — exactly as it already does for RAG embedding failures. No regex heuristics; no new classification logic in the gate itself.

What does NOT arm (unchanged policy):

• Local stdio MCP spawn failures (missing binary, connection refused) — never reach enrichConnectError.
• 4xx client errors (400/401/403) — wrapped in *StatusError for structured access but RetryableHTTPStatus returns false → fail promptly.
• Auth paths (oauthDeclined, authorizationRequired) — handled by their own early-return paths before the status branch; unaffected.
• lifecycle.ErrServerUnavailable, ErrTransport, ErrAuthRequired, ErrInitTimeout, ErrSessionMissing — the gate classifier explicitly excludes all of these.

Deferred:

• A2A pacing: the agent-card resolver does not cleanly expose HTTP status in its error chain; deferred to a follow-up.
• LSP crash-loop pacing: lifecycle.ErrServerCrashed is produced only inside lspSession.Wait() which flows to the supervisor's internal watcher, not to supervisor.Start(). The gate never sees it via the current error propagation path; deferred.

Changed files

File	What changed
pkg/tools/mcp/remote.go	enrichConnectError: wrap on status alone (not gated on a non-empty body); forwards Retry-After via a minimal synthetic *http.Response
pkg/tools/mcp/oauth.go	oauthTransport captures the raw Retry-After header; new lastServerErrorSnapshot() reads status/message/Retry-After together under one lock
pkg/tools/startable_backoff.go	Classifier doc comment: complete excluded-sentinel list (adds ErrInitTimeout, ErrSessionMissing), notes the deferred LSP crash-loop path
pkg/tools/mcp/remote_test.go	8 tests: 503/429/500 → retryable *StatusError; 403 → non-retryable; empty-body 503/429 still arm; Retry-After present/absent; end-to-end TestBackoffGate_* driving a real NewRemoteToolset through tools.StartableToolSet.TryStart
pkg/tools/startable_backoff_test.go	4 tests: lifecycle sentinels (ErrServerUnavailable, ErrTransport, ErrAuthRequired) do NOT arm the gate; 4xx StatusError does NOT arm the gate
docs/tools/mcp/index.md	Startup-failure note: remote MCP 5xx/429 now paced
docs/tools/lsp/index.md	Startup-failure note: crash-loop pacing not yet in place; supervisor policy is the throttle

[aheritier]

🤖 Automated implementer agent — this comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Reviewed at head 608a0991 (this branch is stacked on #4062 and was cascade-rebased when that branch rebased onto a fresher main; content is unchanged by that rebase — single commit, same diff). This PR extends the #4060 backoff gate to remote MCP HTTP errors.

Since this stacked PR only runs a subset of CI (6 checks — canonical-check, link-check, llms-txt-check, markdownlint, pa11y, validate-upstream, all passing; the full build-and-test/lint/windows-tests jobs appear scoped to PRs targeting main), I re-validated locally on the current head:

go build ./pkg/tools/... ./pkg/tools/mcp/...                         # clean
go test ./pkg/tools/... ./pkg/tools/mcp/... -count=1                  # all green
golangci-lint run ./pkg/tools/ ./pkg/tools/mcp/                       # 0 issues

mergeable=MERGEABLE; mergeStateStatus=BLOCKED solely on the missing required approving review.

What this fixes

Remote MCP servers returning 503/429/5xx during the initialize handshake previously triggered a fresh connect attempt on every agent turn (the #4060 burst pattern, extended to MCP). enrichConnectError (pkg/tools/mcp/remote.go:205) now wraps the HTTP status captured by oauthTransport in modelerrors.WrapHTTPError, so retryable responses surface as *StatusError and arm the existing StartableToolSet backoff gate.

Verified

Wrap happens on status alone, not gated on a non-empty body. Many rate-limit/load-balancer responses carry no body. enrichConnectError (remote.go:215) checks status != 0 only; TestEnrichConnectError_EmptyBodyStatusStillArms (remote_test.go:1218) drives a real httptest.Server returning an empty-body 503/429 and confirms the *StatusError still arms.

4xx does not arm. TestEnrichConnectError_NonRetryableStatusDoesNotArm (403 → *StatusError wrapped but RetryableHTTPStatus false) and TestStartBackoffRetryable_4xxStatusDoesNotArm (classifier-level) both pass.

Retry-After is honored and thread-safe. oauthTransport (pkg/tools/mcp/oauth.go) now captures the raw Retry-After header alongside status/body; lastServerErrorSnapshot() (oauth.go:912, 100% covered) reads all three under a single lock so a concurrent request on the same transport (e.g. a standalone SSE probe racing the initialize call) can't have its Retry-After paired with a different response's status. TestEnrichConnectError_RetryAfterHonoured / _NoRetryAfterHeaderLeavesZero cover both the present and absent cases.

End-to-end coverage, not just the enrichConnectError unit boundary. TestBackoffGate_RemoteMCPRetryableStatusPacesReconnect / _NonRetryableStatusFailsPromptly (remote_test.go:1318, :1371) build a real *mcp.Toolset via NewRemoteToolset (matching production wiring), wrap it in tools.NewStartable, and drive TryStart against a mock server — proving the whole chain (enrichConnectError → Toolset.Start → supervisor.Start → the gate) stays intact, and that a 403 reaches the server every turn with no pacing.

Classifier scope is precise and honestly documented. startBackoffRetryable (pkg/tools/startable_backoff.go:35) arms only on *modelerrors.StatusError. The doc comment explicitly excludes lifecycle.ErrServerUnavailable, ErrTransport, ErrAuthRequired/ErrCapabilityMissing, ErrInitTimeout, ErrSessionMissing — each has a dedicated non-arming test (startable_backoff_test.go:763,778,792). Notably, the doc comment and docs/tools/lsp/index.md are both explicit that lifecycle.ErrServerCrashed-based LSP crash-loop pacing is not yet wired (the sentinel is produced only in the supervisor's internal watcher goroutine, not surfaced by supervisor.Start()) — this PR does not overclaim behaviour it doesn't deliver, and defers it honestly rather than shipping dead-code coverage for it (an earlier draft of this PR did include that dead arm; it was caught in review and removed).

Coverage: go tool cover -func on ./pkg/tools/mcp/ shows both enrichConnectError and lastServerErrorSnapshot at 100%.

Deferred (documented, not silently dropped)

• A2A pacing — the agent-card resolver doesn't cleanly expose HTTP status.
• LSP crash-loop pacing — needs ErrServerCrashed propagated through supervisor.Start() first.

PR body/description updated to reflect the full accumulated diff (oauth.go changes, Retry-After handling, and all 8 new MCP tests were missing from the original description; the "Changed files" table and Design section now match the current commit).

No blocking findings. Filing as a comment rather than a formal approval only because I authored the PR and GitHub blocks self-approval for the author, matching the pattern on #4062.