serve: enforce the permission firewall on every REST route (#655)

[padak]

Fixes #655.

The gap

PermissionEngine was built only in the Typer callback (cli.py), so a persisted permissions set --mode deny policy — and both --deny-writes and --deny-destructive — protected the CLI process and nothing else. kbagent serve exposed all 236 routes, DELETE /storage/buckets and POST /token/{p}/delete included, behind a single all-or-nothing bearer token.

#677 built the enforcement machinery — an engine on app.state, a PermissionDeniedError → HTTP 403 handler, and the require_permission dependency — and wired it to three /auth/* routes. This PR adds the missing half: coverage.

How

One app-level dependency (FastAPI(dependencies=[...]), so no router can be added outside it) runs on every request, looks the matched route up by (method, path template) in a central table, and calls the same PermissionEngine.check_or_raise the CLI uses. FastAPI puts the matched route into request.scope["route"] before dependencies resolve, so the template — /storage/buckets/{project}, not the concrete URL — is what the policy is keyed on. Verified at runtime, not assumed.

Why one table instead of 226 per-route decorators:

1. One auditable screen. "What can a caller still do under --deny-destructive?" is answered by reading one file, not thirty routers.
2. Fail-closed by construction. A route with no entry is refused, not silently allowed — the failure mode of a forgotten annotation is a loud 403, never an open door.

The per-route require_permission(...) form still wins where it is declared: such a route is skipped by the table lookup (recognised via a PERMISSION_DEPENDENCY_MARKER attribute). That is what keeps #677's three /auth/* routes — and the test-only probe routes in tests/test_server_permissions.py, registered after create_app — working unchanged.

With no policy configured, nothing changes. is_allowed returns True for everything, so the default install behaves exactly as before.

Also in scope

• GET /permissions/show (the LOW ask in serve: missing mirrors — describe-batch (inline payload), unload-table; job run lacks idempotency_key #657, and serve: PermissionEngine firewall is not enforced on REST routes #655's option-2 discoverability point). Reports the effective policy — the persisted block already merged with the --deny-* flags the daemon was launched with, which a REST caller can neither see nor change. Reachable under any policy (the same anti-lockout rule the CLI has), still requires the bearer token. Deliberately no write counterpart: letting a bearer token widen the policy that constrains it would make the firewall self-defeating, so permissions set / reset stay terminal actions on the host.
• Two new serve-only operations, ai.chat and workspace.sql-improve, both write: neither touches Keboola, but both spawn a local claude/codex/gemini process on the host, exactly like the already-write agent.prompt-improve. Added to SERVE_ONLY_OPERATIONS so the command-sync gate does not read them as dead keys.
• The false comment serve: PermissionEngine firewall is not enforced on REST routes #655 called out. permissions.py's http.* block claimed "The serve's own routes enforce their own permissions on top." It is true now, and the comment says precisely since when.

Classification notes

• Every DELETE route maps to a destructive- or admin-class operation; a test asserts it.
• Eight POSTs map to read operations (/flows/validate, /lineage/show, /kai/ask, …) — POST because the request needs a body, not because it mutates. Explicit allowlist in the test, so a new mutating route classified read fails.
• GET /branches/{project}/merge-url maps to the write-class branch.merge. That is CLI parity, not a slip: kbagent branch merge only ever produces a URL too.
• Granularity caveat, documented: POST /semantic-layer/items/{kind} covers metric/dataset/… in one route, so it maps to the collapsed parent key semantic-layer.add, not semantic-layer.add.metric. A policy naming only a leaf key is enforced on the CLI but not over REST — name the parent or a cli:* category to cover both.
• Nine bootstrap paths are never checked (/health/ping, /health/auth-info, /ui-config, /docs, /redoc, /openapi.json, /docs/oauth2-redirect, and the SPA shell / + /index.html). A locked-down server must still be able to say who it is, or a client cannot tell a policy refusal from a dead process.

Testing

make check green: 6148 passed, 12 skipped. ty clean (the one remaining diagnostic is the pre-existing scripts/hatch_build.py unresolved import). Lint, format, command-sync, version-gates, sentinel-guards, error-codes and the endpoint-reference gate all pass; docs/web-server-endpoints.md regenerated and committed.

31 new tests in tests/test_server_route_permissions.py (new file, so no conflict with #681's edits to test_server_permissions.py):

• Completeness, both directions — every live route is mapped, exempt, or inline-guarded; no table entry matches a dead route; every table value is a real OPERATION_REGISTRY key. This is what keeps the runtime fail-closed branch unreachable in a released build.
• Enforcement on real routes — cli:destructive denies DELETE /storage/buckets/{project} while GET /projects stays 200; --deny-writes denies POST /jobs/{p}/run; an exact token.delete pattern works; mode=deny blocks an unlisted read but still serves /health/*.
• Fail-closed — a route registered without a table entry answers 403 naming ROUTE_OPERATIONS.
• /permissions/show — clean, persisted, effective-with-flags, inert patterns, reachable under total deny, still 401 without the token.
• Verb/risk agreement — the three structural checks above.

Mutation-checked: commenting out the app-level dependency fails exactly 5 of them, so they are load-bearing rather than decorative.

A trap worth flagging for review

FastAPI 0.137 stopped flattening include_router eagerly — app.routes holds 35 lazy _IncludedRouter proxies instead of the 236 routes they stand for, and nothing materialises them (not app.openapi(), not TestClient startup). Request handling is unaffected, but a completeness test that walked app.routes naively would audit four routes, find nothing wrong, and pass. _iter_api_routes recurses through original_router; the docstring says why, because this is exactly the false-pass shape a coverage test must not have.

Merge-order notes

• serve: honor the root-level --config-dir, and fix two stale permissions comments #681 (open) also edits src/keboola_agent_cli/permissions.py. Both PRs rewrite the same three-line http.* comment (it fixes the false claim; this makes the claim true). Expect a small conflict there — resolve in favour of whichever lands second, keeping this PR's wording if that is this one.
• serve: missing mirrors — describe-batch (inline payload), unload-table; job run lacks idempotency_key #657, if implemented, adds REST routes and will need matching ROUTE_OPERATIONS entries. The completeness test names each missing route, so the failure is self-explaining rather than mysterious.

No version bump, no changelog entry (per CONTRIBUTING: those belong to the release PR). New behaviour is gated with the literal (since vNEXT) placeholder on every doc surface; context.py carries no version tag, matching the precedent set in #681.

Doc surfaces updated (convention #17)

docs/web-server.md (new "The session firewall applies to every route" section, plus two stale paragraphs that asserted the gap), docs/web-server-endpoints.md (regenerated), CLAUDE.md, plugins/kbagent/skills/kbagent/references/gotchas.md (the "A deny policy does NOT firewall the whole REST surface" entry was live and is now inverted, with the 0.90.1-and-older behaviour kept for readers on those versions), commands-reference.md, and commands/context.py's AGENT_CONTEXT. No CLI command added, renamed, or removed, so keboola-expert.md and SKILL.md need no change.