v0.8.15: ci improvements, trigger machine resizing, connectors fixes - #7217
Conversation
waleedlatif1
commented
Aug 28, 2026
- improvement(api): retire route contracts that no route serves (improvement(api): retire route contracts that no route serves #7206)
- improvement(ci): bound docker layer caches and right-size ten runners (improvement(ci): bound docker layer caches and right-size ten runners #7210)
- improvement(background): right-size the two knowledge task machines (improvement(background): right-size the two knowledge task machines #7212)
- improvement(ci): scan CodeQL PRs at the promotion boundary, refresh main daily (improvement(ci): scan CodeQL PRs at the promotion boundary, refresh main daily #7213)
- improvement(ci): bound the local Turborepo cache (improvement(ci): bound the local Turborepo cache #7214)
- fix(connectors): honor provider retry deadlines (fix(connectors): honor provider retry deadlines #7211)
* fix(api): retire route contracts that no route serves The staging integ alarm fired because the session knowledge-document inline-create check got a 405. #7179 moved tool operations in process and deleted the routes that only existed to serve them, but `createKnowledgeDocumentsContract` kept declaring `POST /api/knowledge/[id]/documents` — a path whose surviving GET/PATCH make Next.js answer POST with 405 rather than an honest 404. Nothing in the repo called it: the KB UI creates documents through the presigned upload flow, and the capability itself is unaffected because `knowledge_create_document` reaches the same use case in process. Audited all 1125 contracts for the same drift. It was the only one whose path resolves to a live route missing the declared method; 259 others declare paths of routes that were deleted outright, which 404 honestly and are left alone. - Drop the create-documents route contract for plain `params`/`body` schemas plus a named response schema, so nothing declares an endpoint we do not serve. The schemas stay in the contracts tree next to the siblings they share (`documentDataSchema` is used by the v2 contracts, and `createKnowledgeDocumentsBodySchema` already backed the in-process operation). - Delete four contracts with no consumer at all — both TTS contracts, docusign, and mistral. Their handlers own better schemas: TTS dispatches by `toolId` with eight per-provider schemas instead of one passthrough superset, and mistral bounds `pages` by the OCR request policy. crowdstrike and windchill look similar but are load-bearing (schema and derived types are imported by live code), so they stay. - Add `check:api-contract-routes`, picked up automatically by `run-audits`. `check:route-verbs` scans routes to contracts, so a contract whose route method was deleted is invisible to it — verified it passes clean against the exact regression this catches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(api): read contracts by import, and retire two more stale declarations Greptile flagged the audit's brace counter as blind to braces inside strings, template literals, regexes and comments. It was, but the bigger problem was that a text scan can only see contracts whose `method`/`path` are inline literals — the 70-plus built through `definePostSelector(path, …)` and friends were never checked at all. Comparing raw `defineRouteContract(` occurrences against parsed ones showed the scanner silently skipping declarations. Read the contracts by importing each contract module and inspecting its exported objects instead, the way `check-route-verbs.ts` already resolves the contract behind a route. Contract modules are pure Zod so importing them is safe; route files stay a static scan because importing one drags in `@sim/db`, auth and `next/server`. Barrels re-export the same object, so entries are keyed by identity. Coverage goes from 1125 contracts to 1283. That immediately surfaced two more instances of exactly what this PR retires. `/api/tools/confluence/page` kept its `PUT` and `DELETE` contracts after #7179 reduced the route to the selector `POST`, so both declared verbs the live route answers with 405. Neither is fetched — `lib/internal/confluence/execute-tool.ts` is the only consumer — so they become plain schemas like the knowledge one, and `executeOperation` now delegates to a schema form rather than growing a second pattern beside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#7210) The Blacksmith sticky disks backing our docker layer caches had no eviction policy. setup-docker-builder skips pruning entirely unless max-cache-size-mb is set, and BuildKit's own GC is time-based only (8 days unused), so on a repo that builds this often nothing ever aged out: app.Dockerfile/linux-amd64 hit 351 GB within a day of being created, and realtime — an image under 300 MB — sat at 249 GB. Layer caches alone were 920 GB across ten disks. Cap them per image via a cache_mb matrix field, alongside the bs_runner field that already encodes per-image sizing. The app image keeps 100 GB (several generations over its working set of ~34 layers plus monorepo apt/bun cache mounts); everything else takes the 25 GB default, which is still 4x the tightest working set in the matrix (pii, whose spaCy models are ~2.2 GB). The fallback lives in the composite action rather than an input default, because an unset matrix key arrives as the empty string and would bypass a default — silently restoring unbounded growth on any row that forgot the field. Runner sizes follow measured CPU and memory percentiles over 30 days: - CodeQL splits per language. javascript-typescript peaks at 19.5 GB so it stays on 8 vCPU; actions peaks at 1.3 GB and averages 22% CPU over a 39s median run, and drops to 4 vCPU. - The pii and realtime image builds drop to 2 vCPU. Both already ran on 8 vCPU earlier in the window, so the 8->4 step is measured, not modelled: realtime went 52s -> 51s and pii 24s -> 27s. - Five desktop release jobs drop to 2 vCPU. They peak under 0.4 GB and finish in 4-13s. None of these sit on their group's critical path — each has 190-275s of slack behind an app build that dominates it — so wall-clock is unchanged. Those five desktop jobs also hardcoded a Blacksmith label with no CI_PROVIDER fallback, against the invariant stated at the top of ci.yml. In GitHub break-glass mode they would have sat in `queued` forever; they now fall back like every other job. Left alone deliberately: the 16 vCPU app builds (memory-bound, and 16 vCPU measured 2.1x faster and 6% cheaper than 8 vCPU), Lint and Test (CPU-bound, 62% of the run above 80%), and the push-path Build App (build-amd64 has no needs:, so it is what stops a migration applying for a build that cannot ship).
…7212) Both knowledge tasks reserved machine presets well above their measured ceilings. Sized each from production telemetry on both memory and CPU: - knowledge-connector-sync: large-2x -> large-1x. Peak sampled RSS 2.6 GB and peak 1.4 vCPU, so 8 GB/4 vCPU keeps ~3x memory and ~2.8x CPU headroom against a preset that reserved 16 GB. - knowledge-process-document: large-1x -> medium-2x. Peak sampled RSS 902 MB and peak 1.2 vCPU, with no document exceeding 2 GB, so 4 GB/2 vCPU keeps ~4x memory and ~1.7x CPU headroom. CPU figures are core-normalized (OTel process.cpu.utilization divides by cores available), so neither task loses headroom it was actually using and neither can be throttled by the smaller preset. No retry or concurrency semantics change.
…ain daily (#7213) Feature PRs land on staging and are ~90% of PR scan volume (90 of the last 100 PRs target staging, 4 target main). Every one of them is scanned again — against the exact tree being promoted — when the staging->main PR opens, so restricting PR scans to main defers the signal to the promotion boundary rather than dropping it. No ruleset or branch protection requires a CodeQL check, and the alert view is fed by the push-to-main and scheduled analyses, not by PR runs. Deliberately a branch cut rather than an activity-type cut. Dropping `synchronize` would have cut a similar share of runs, but it scans a PR's first commit and never its final state — backwards, since review fixups land in later pushes. The scheduled scan moves from weekly to daily. Pushes to main are rare, so with PR scans limited to main the default-branch alert view leans on the cron more than it used to, and a week is too long to leave it stale. It also reseeds the overlay-base database that PR runs restore from: that cache key embeds the CodeQL bundle version, so a bundle bump invalidates it, and an unused Actions cache is evicted after 7 days. Also records, in codeql-config.yml, why the obvious speed-up is a trap: adding `queries:`/`packs:`/`query-filters:` trips OverlayDisabledReason.NonDefaultQueries and permanently disables overlay analysis, trading a documented up-to-10x win on the extraction phase (~53% of a run) for a few percent off the query phase.
Turborepo's local cache eviction is opt-in until 3.0, so without cacheMaxAge and cacheMaxSize the filesystem cache grows forever. In CI each cache dir lives on a Blacksmith sticky disk that is mounted many times a day, so it never idles long enough for Blacksmith's own 7-day inactivity purge to fire, and one cache-missing app build writes a ~400 MB artifact. The build cache disk reached 206 GB over 43 days — roughly 4.8 GB/day of sediment — at ~$0.51/GB-month. Size is the real bound; age is hygiene. A cache hit only happens when a task's input hash is unchanged, which recurs within hours, not weeks, so nothing written days ago can ever be read again — the sibling PR-keyed disk does the same job in 30 GB. The cache still earns its keep: the app build hits ~17% of the time and a hit saves ~7 minutes, so this is a ceiling, not a removal. That distinguishes it from the Turbopack persistent cache, which was removed because it measured 3.2x SLOWER; this one is measurably faster, just unbounded. Set in turbo.json rather than per-step env vars so there is one source of truth and turbo validates it. Verified against the installed 2.9.14: both keys are in its schema, turbo parses them, and the task hash is byte-identical with and without them, so enabling eviction does not invalidate the existing cache.
* fix(connectors): honor provider retry deadlines * fix(connectors): validate retry response lifecycles
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Greptile SummaryThis release improves CI cache bounds and runner sizing, resizes two knowledge background tasks, narrows and refreshes CodeQL scanning, removes contracts for retired routes, hardens Atlassian discovery validation, and propagates provider retry deadlines into connector scheduling.
Confidence Score: 4/5The PR appears safe to merge, with a non-blocking cleanup needed in the new route-audit script's logging and error handling. The changed runtime paths preserve internal schema validation and bounded connector backoff behavior; the only accepted issue is that the new developer audit command bypasses repository-standard diagnostics utilities. Files Needing Attention: scripts/check-api-contract-routes.ts
|
| Filename | Overview |
|---|---|
| apps/sim/lib/knowledge/documents/utils.ts | Centralizes bounded retryable HTTP errors, response cleanup, and retrieval of validated retry delays through error cause chains. |
| apps/sim/lib/knowledge/connectors/sync-engine.ts | Propagates provider retry deadlines into connector scheduling without weakening the existing failure backoff or one-day ceiling. |
| apps/sim/lib/internal/tool-operations/parse-contract-input.ts | Generalizes internal request parsing so retired HTTP operations can validate against bare schemas while preserving contract callers. |
| scripts/check-api-contract-routes.ts | Adds a route/contract consistency audit, but its diagnostics bypass required shared logging and error-normalization utilities. |
| .github/workflows/ci.yml | Bounds Docker caches and adjusts runner sizes across development, release, and desktop jobs. |
| .github/workflows/codeql.yml | Moves PR scanning to the main promotion boundary, refreshes main daily, and sizes runners by analyzed language. |
| apps/sim/lib/atlassian/discovery.ts | Validates accessible-resource response shape and distinguishes malformed responses from credentials with no site grants. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Provider[Connector provider] --> Fetch[Retrying fetch layer]
Fetch --> Error[Retryable HTTP error]
Error --> Delay[Validated retryAfterMs]
Delay --> Sync[Connector sync failure handler]
Sync --> Backoff[Max of failure ladder and provider delay]
Backoff --> Schedule[nextSyncAt, capped at one day]
Reviews (1): Last reviewed commit: "fix(connectors): honor provider retry de..." | Re-trigger Greptile