[awf] network: isolation/topology mode + Tailscale → Squid 503 TCP_TUNNEL:HIER_NONE on api.githubcopilot.com egress

[jeremyhayes]

Problem

In network-isolation / topology mode, when a workflow also brings up Tailscale, the api-proxy's egress CONNECT to api.githubcopilot.com:443 fails at Squid with 503 TCP_TUNNEL:HIER_NONE. The request never leaves the sandbox, so the Copilot engine reports Failed to get response from the AI model; retried 5 times … Last error: 503 Service Unavailable and the agent job fails.

The same workflow works on the older host-access stack, and a non-Tailscale workflow works in the same isolation mode — so it's the isolation-mode × Tailscale combination, not the Copilot API, the model, or the request.

Where observed

• Workflow: github/copilot-code-review → datadog-alert-triage (Copilot engine, claude-opus-4.8). It brings up Tailscale via github/setup-tailscale to reach 5 Splunk *.ts.net MCP servers, plus Datadog/Sentry/Kusto/GitHub MCP.
• Reproduced on gh-aw v0.82.15, v0.83.1, v0.83.4 (api-proxy 0.27.37 → 0.27.42, Copilot CLI 1.0.71 → 1.0.75). Works on main's stack (gh-aw v0.81.6, api-proxy 0.27.11, host-access mode).

Evidence

Squid access log — failing run (api-proxy 0.27.37): every CONNECT to the Copilot API fails:

api.githubcopilot.com:443 -:- 1.1 CONNECT 503 TCP_TUNNEL:HIER_NONE api.githubcopilot.com:443 "-"

→ 56/56 CONNECTs = 503.

Same workflow, working run (api-proxy 0.27.11, host-access): 42/42 CONNECTs = 200.

Resolved firewall config diff (audit/awf-resolved-config.json):

	failing (new)	working (old)
networkIsolation	true	false
enableHostAccess	false	true

api-proxy OTEL span (failing): GET /models → http.response.status_code: 503 (the proxy just propagates the Squid CONNECT failure; the fatal one is the chat request).

Copilot harness classification: failureClass=partial_execution, isModelNotSupportedError=false, isCAPIQuotaExceededError=false, isAuthError=false.

Environment: Squid 6.14; nameserver 168.63.129.16 (Azure DNS).

Run IDs (github/copilot-code-review Actions):

• Fail — v0.82.15 / 0.27.37: 30401644133
• Fail — v0.83.4 / 0.27.42: 30397968304
• Fail — v0.83.1 / 0.27.38: 30390759136, 30394229055
• Work — main / 0.27.11: 30393390874, 30402440480

Isolation matrix (what rules causation in/out)

firewall mode	Tailscale up?	api.githubcopilot.com egress
host-access (0.27.11)	yes	✅ 200
isolation/topology (0.27.37)	no — github/copilot-metrics project-triage, claude-opus-4.8, succeeds daily	✅
isolation/topology (0.27.37–0.27.42)	yes — our workflow	❌ 503 HIER_NONE

This rules out: the Copilot API (never receives the request), the model / retirement (isModelNotSupportedError=false; opus-4.8 serves fine elsewhere), request size (the CONNECT fails before any payload), and repo/token (works on host-access with identical secrets). The only differentiator is isolation-mode + Tailscale.

Suspected root cause

In isolation/topology mode the agent + api-proxy egress traverses Squid with no host route (per #6657). When Tailscale comes up it rewrites routing/DNS; in the isolated topology this appears to black-hole the route/resolution to api.githubcopilot.com, so Squid has no forwarding path for the CONNECT and returns HIER_NONE. In host-access mode the host route sidestepped the collision. (HIER_NONE is a forward/resolve failure, distinct from an ACL TCP_DENIED.) Maintainers can confirm the exact interaction — e.g. whether Tailscale is capturing the default route or the Azure DNS (168.63.129.16) path used by Squid.

Related

• [awf] network: MCP gateway blocked despite topologyAttach — recommendation loop #6670 (open) — same subsystem: Squid ACL doesn't auto-allow topologyAttach hosts (internal MCP gateway blocked). Likely a sibling gap in topology-mode egress/ACL handling.
• fix: allow --enable-host-access with --network-isolation (topology mode) #6657 (merged) — host-access × network-isolation; documents that in topology mode the agent runs on an internal Docker network with no host route.

Impact

Blocks upgrading github/copilot-code-review's opus-based FR/alert-triage fleet past the current host-access stack — every newer stack tested 503s. Any Tailscale-using workflow in isolation mode is likely affected.

Possible mitigations to validate

• Exclude allowlisted external egress domains (e.g. *.githubcopilot.com) from Tailscale's route table / DNS interception when isolation mode is active.
• Bring Tailscale up so it doesn't capture the default route / DNS resolver Squid uses for egress.
• Provide a host-access egress path for the api-proxy in topology mode (cf. fix: allow --enable-host-access with --network-isolation (topology mode) #6657).

---

Filed after an end-to-end investigation; happy to share full firewall log artifacts or run a controlled Tailscale-removed repro on request.

[jeremyhayes]

Reopening — #6705's DNS fix is necessary but not sufficient; the HIER_NONE egress failure persists.

I validated firewall v0.27.43 (which contains #6705, 7b6a767) against the workflow that originally hit this — github/copilot-code-review's datadog-alert-triage (network-isolation / topology mode, Tailscale up for the Splunk MCP servers, Copilot engine / claude-opus-4.8).

The DNS fix runs correctly — the agent log shows it doing exactly what #6705 intended:

[WARN] Network-isolation: removing 1 non-portable DNS server(s) ... : 168.63.129.16
[WARN] Network-isolation: no portable DNS servers remain after filtering; falling back to 8.8.8.8, 8.8.4.4.

and Squid comes up with Adding nameserver 8.8.8.8 / 8.8.4.4.

But egress still fails identically. Every CONNECT to api.githubcopilot.com returns 503 TCP_TUNNEL:HIER_NONE (80/80 this run), dest server field empty (-:-) — i.e. DNS still never resolves:

… 172.30.0.30:42410 api.githubcopilot.com:443 -:- 1.1 CONNECT 503 TCP_TUNNEL:HIER_NONE api.githubcopilot.com:443 "-"

and the agent dies with the same Failed to get response from the AI model … Last error: 503 Service Unavailable (isModelNotSupportedError=false).

Why the DNS swap doesn't help. The root cause #6705 itself described is the Tailscale policy route — "if an exit-node / accepted-subnet route covers 0.0.0.0/0, Docker bridge traffic on awf-ext follows from all lookup 52 through the tunnel." That capture covers the default route, so from the isolated bridge 8.8.8.8 is just as unreachable as 168.63.129.16 — changing which DNS server Squid points at can't restore egress when the route to any off-bridge address is black-holed (and the same applies to the eventual CONNECT to the Copilot API's resolved IP).

So #6705 fixed the DNS-server layer; the routing-capture layer is the remaining blocker. A fix likely needs to keep the awf-ext bridge egress — at minimum to the allowlisted domains + its DNS — out of the Tailscale-captured table 52 path (e.g. a higher-priority policy rule for the bridge subnet, or scoping Tailscale's accepted routes), rather than only swapping the DNS server.

Environment: firewall image 0.27.43, networkIsolation: true, enableHostAccess: false, gh-aw v0.84.2, Tailscale up. Happy to share the full firewall log artifacts or run a controlled repro on request.

[jeremyhayes]

@lpcox I don't have permission to reopen here, but the original issue here is unresolved.

[lpcox]

@jeremyhayes seems Tailscale's installing a default/accepted route that captures the awf-ext bridge egress. Route to public DNS also captured. Could we try this?

• Do not configure a Tailscale exit node.
• Use --accept-routes=false if the *.ts.net MCP servers are ordinary tailnet peers.
• If subnet routes are required, ensure they do not include 0.0.0.0/0.
• Alternatively, use Tailscale userspace networking/SOCKS5 only for private MCP connections.

Would really rather not run awf w/ CAP_NET_ADMIN and ask it to add new host policy-routing rules. Can you re-run w/ tailscale enabled but no default/accepted routes to confirm if table-52 route capture is the cause? Thanks!

[jeremyhayes]

@lpcox — ran the --accept-routes=false validation you asked for. Short version: disabling route acceptance fixes the isolation-mode egress failure and preserves our private-MCP-over-Tailscale connectivity, which points the durable fix at the isolation-mode egress path rather than DNS. Mechanism below; happy to be wrong on any of it. (I'll keep our infra specifics out of this public thread — glad to share the full dumps with you directly.)

Setup

Our triage workflow on gh-aw v0.84.2 / firewall 0.27.43 (i.e. with the #6705 DNS fix), private MCP servers (reached over Tailscale) left in. github/setup-tailscale → tailscale/github-action hardcodes --accept-routes (no opt-out, and passing --accept-routes=false at initial up fails with exit 2), so I let Tailscale come up normally, then reconfigured with tailscale up --reset --accept-routes=false before the firewall sandbox starts — from the sandbox's point of view, equivalent to coming up with acceptance off.

Result 1 — egress recovers

The api-proxy's CONNECT to the model endpoint goes from 503 TCP_TUNNEL:HIER_NONE (on 0.27.43 with acceptance on) to 200 TCP_TUNNEL:HIER_DIRECT, and the agent job runs to completion.

Result 2 — private MCPs still reachable

All of our Tailscale-reached MCP backends still complete a TLS handshake and respond (HTTP 401 — reached, auth-gated) from both the host and a Docker-bridge container approximating the sandbox path.

Why both hold

--accept-routes is all-or-nothing, and it drags the tailnet's advertised subnet routes into policy table 52 (rule 5270: from all, consulted before main). One of those advertised subnets overlaps the runner's own default-gateway subnet, so the firewall's isolation-mode egress (squid on the internal: true bridge → host → gateway) gets pulled into tailscale0 and black-holed. Turning acceptance off removes only those advertised subnet routes; the tailnet mesh peer routes — which is what our private MCPs actually use — stay put, so MCP connectivity is unaffected. Host-access mode egressed normally and was never exposed, which lines up with this being an isolation-topology × host-routing interaction rather than a resolver issue (#6705 runs fine here but doesn't help).

Possible fix (you'll know the internals better)

In isolation mode, pin the firewall's own egress to main so host policy routing can't capture it — e.g. a higher-priority ip rule (priority < 5270) matching the firewall's egress (by its bridge source subnet, or whatever fwmark you already set) → lookup main. That would make sandbox egress immune to any workflow that rewrites host routing, not just Tailscale. Very open to a better shape if that's the wrong approach.

Happy to walk through the access.log, ip route/ip rule before-and-after, and resolved firewall config with you directly (not here) — just point me at the best channel.