perf(db): optimize recurring query paths

[waleedlatif1]

Summary

• replace offset-based log export pagination with deterministic keyset reads and bounded streaming hydration
• combine related billing aggregates and batch repeated workspace and custom-block reads
• add query-shape-matched online indexes, remove catalog-identical redundant indexes, and tune maintenance thresholds for an append-heavy ledger

These changes preserve existing authorization, filtering, ordering, aggregation, and response contracts. Index changes intentionally balance targeted read improvements against their write and storage cost.

Type of Change

• Improvement

Testing

• focused regression suites for logs, billing, knowledge, custom blocks, VFS, and workspaces
• bun run type-check
• bun run lint
• bun run check:audits
• bun run check:migrations origin/staging
• Drizzle schema and migration integrity checks

Checklist

• Code follows project style guidelines
• Self-reviewed the changes
• Tests were added or updated and pass
• No new warnings were introduced
• Database changes use one new backward-compatible migration

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 23, 2026 6:47pm

[cursor]

PR Summary

High Risk
Changes paid usage calculation (ledger + daily refresh) and ships a production migration that rebuilds indexes concurrently and retunes usage_log autovacuum. Incorrect usage math or a failed concurrent index build would affect billing gates and live writes.

Overview
Cuts load on several hot read paths without changing auth, filters, or response contracts.

Logs export now pages with a (startedAt, id) keyset instead of OFFSET, streams CSV with highWaterMark: 0, and materializes execution data in bounded chunks so cancelled downloads stop pulling pages.

Billing reads ledger totals and daily-refresh consumption from one snapshot via computeBillingPeriodUsageWithDailyRefresh. Paid personal/org usage still clamps at zero after refresh; unpaid/reporting paths stay ledger-only.

Batching: public workspace listing hydrates a page of details and member counts in one query; custom-block input hydration is concurrency-capped and passes source workspaceId; copilot VFS loads custom blocks once per materialize (retry on failure). Knowledge docCount counts knowledgeBaseId to match a new partial index.

A migration adds query-shaped indexes (documents, connector sync logs, table views, workflows), drops duplicate unique-key indexes, and lowers usage_log autovacuum insert/analyze scale factors.

^{Reviewed by Cursor Bugbot for commit 72eb8b6. Configure here.}

[greptile-apps]

Greptile Summary

The PR optimizes recurring database reads through keyset log-export pagination, bounded execution-data hydration, combined billing aggregation, batched workspace and custom-block reads, and query-matched indexes.

• Log exports now fetch deterministic 100-row pages while yielding hydrated CSV rows under pull backpressure.
• Billing and workspace-related paths consolidate repeated database operations.
• The migration adds targeted online indexes and removes redundant indexes while preserving deploy-safe migration behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported one-query-per-row amplification is fixed by bounded 100-row keyset pages, 20-way hydration chunks, and deferred next-page reads.

Important Files Changed

Filename	Overview
apps/sim/app/api/logs/export/route.ts	Replaces offset pagination with deterministic 100-row keyset reads, bounded hydration, pull-driven row delivery, and cancellation handling.
apps/sim/app/api/logs/export/route.test.ts	Covers authorization, cursor precision and ordering, bounded hydration, page-level backpressure, and cancellation.
apps/sim/lib/billing/calculations/usage-monitor.ts	Consolidates ledger and daily-refresh calculations while retaining paid-plan clamping and organization-member bounds.
apps/sim/lib/billing/credits/daily-refresh.ts	Combines billing-period ledger totals and bounded daily-refresh aggregation into fewer database reads.
packages/db/migrations/0303_colorful_gressill.sql	Adds query-shape-matched indexes, removes redundant indexes, and adjusts maintenance settings using deploy-safe concurrent operations.
packages/db/schema.ts	Aligns the Drizzle index definitions and table maintenance configuration with the migration.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[CSV consumer pull] --> B[Fetch up to 100 logs by keyset]
  B --> C[Hydrate up to 20 execution payloads]
  C --> D[Yield CSV rows individually]
  D --> E{Page consumed?}
  E -- No --> D
  E -- Yes, full page --> F[Advance startedAt and id cursor]
  F --> B
  E -- Yes, partial page --> G[Close stream]

Loading

_{Reviews (5): Last reviewed commit: "fix(logs): guard export stream cancellat..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 396884f. Configure here.}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 72eb8b6. Configure here.}