UN-3883 [FEAT] Cut dashboard metrics cron DB load: narrower source windows, monthly from daily, two new indexes

[kirtimanmishrazipstack]

What

Three changes to the dashboard metrics cron, already reviewed and merged individually as #2255, #2264 and #2265:

• Monthly figures are now added up from the daily figures instead of being recalculated from the raw tables every time.
• Each run looks back 2 days instead of ~2 months. A separate pass at 04:40 UTC looks back 7 days once a day, so a short outage still repairs itself.
• The expensive half of the work runs hourly instead of every 15 minutes. Hourly figures keep their 15-minute cadence; daily and monthly move to hourly at :20.
• Two new database indexes, on the two tables the cron reads most.

Why

The cron was using roughly 55 minutes of database time every 6 hours on production. It re-read between 32 and 62 days of raw data for all 38 organisations, 96 times a day, to produce figures that only change once a month.

Nothing a customer sees changes. Hourly dashboard numbers are still at most 15 minutes old; daily and monthly numbers are now at most an hour old instead of 15 minutes, which is the one deliberate trade.

How

• _rollup_monthly_from_daily replaces the per-org monthly source queries with one INSERT ... ON CONFLICT DO UPDATE over event_metrics_daily for every org. Upsert-only, per the design agreed on UN-3973: a monthly row the daily tier no longer produces is left in place, because a stale total is recoverable with backfill_metrics and a deleted one is not.
• aggregate_metrics_from_sources(tier, source_window_days) is called by three schedule rows. Both kwargs travel on both transports — Beat calls the Django task directly, the PG scheduler goes worker proxy → internal endpoint → the same function.
• Lock keys are one per granularity written, namespaced by source window. ALL takes both, so it genuinely excludes a concurrent hourly run; distinct windows are distinct jobs, so the once-daily reconciliation pass can never be starved by the 15-minute schedule it is never retried after.
• Both indexes are built CONCURRENTLY under atomic = False, with AddIndex confined to state_operations. Each migration asserts the index is valid and has the expected definition before recording itself applied.

Can this PR break any existing features?

The realistic risks, and what bounds each:

• Monthly figures go stale if the daily tier has a gap. Monthly is now only as good as event_metrics_daily. Gaps shorter than 7 days repair themselves on the next reconciliation pass; anything older needs backfill_metrics. Bounded to the current and previous month.
• Rolling the code back past this release breaks aggregation until migration 0006 is reversed — the schedule rows carry a tier kwarg the previous release's signature rejects. Documented in the migration docstring; migrate dashboard_metrics 0005 restores it.
• A pod still on the old image during the rollout receives tier and raises TypeError until it rolls. Self-healing, and no aggregation is lost because the next tick succeeds.
• Both index builds are non-blocking, and both migrations are reversible.

Database Migrations

Four, in three apps. No schema changes to any metrics table.

Migration	What it does
dashboard_metrics/0005_add_reconciliation_task	Adds the 04:40 UTC reconciliation schedule, on both Beat and the PG scheduler
dashboard_metrics/0006_split_aggregation_schedule	Splits the aggregation into two rows by tier
file_execution/0007_wfe_status_created_idx	Index on workflow_file_execution (status, created_at)
workflow_v2/0029_we_created_at_idx	Index on workflow_execution (created_at)

Both index migrations no-op via IF NOT EXISTS if the index was built out of band first, which is the preferred production path — the exact statement is in each migration's docstring.

Env Config

None.

Deploy Steps

Run once, before the first aggregation after deploy:

python manage.py backfill_metrics --days 60 --skip-hourly --skip-monthly

Monthly is now derived from the daily tier, so that tier has to be complete across the rollup window first. --skip-monthly is deliberate: repair daily and let the rollup derive monthly.

Relevant Docs

backend/dashboard_metrics/README.md is updated — schedules, windows, staleness bounds, and the ownership overlap with backfill_metrics.

Related Issues or PRs

Merged into this branch: #2255 (UN-3973), #2264 (UN-3972), #2265 (UN-3974). Parent: UN-3883.

Dependencies Versions

No dependency changes.

Notes on Testing

Ticket	Acceptance criterion	State
UN-3973	Monthly derived from event_metrics_daily, not source tables	Met
UN-3973	Per-run daily source window is 2 days	Met
UN-3973	A once-daily 7-day reconciliation pass exists and is scheduled	Met
UN-3973	Daily and monthly match pre-change values, incl. across a month boundary	Met
UN-3973	Tests cover the month boundary and the reconciliation pass	Met
UN-3972	Index present, indisvalid = t	Met
UN-3972	Non-atomic + CONCURRENTLY, no write-blocking lock	Met
UN-3972	get_documents_processed free of a seq scan on workflow_file_execution	Confirm on prod
UN-3972	get_failed_pages free of a seq scan on workflow_execution	Met
UN-3972	get_recent_activity under 1 s	Flagged — belongs to the (created_at DESC) index descoped in comment 45015
UN-3974	Hourly stays 15 min, daily/monthly go hourly	Met
UN-3974	Hourly figures unchanged, daily/monthly lag ≤ 1 h	Met
UN-3974	Prefilter leaves the Query Insights top 10	Confirm on prod

The two Confirm on prod rows are post-deploy readings against Query Insights, not outstanding work.

Screenshots

Not applicable — no UI change.

Checklist

I have read and understood the Contribution Guidelines.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud

[greptile-apps]

Greptile Summary

The PR reduces dashboard aggregation database load by splitting hourly and daily/monthly schedules, narrowing source windows, rolling monthly metrics up from daily data, and adding supporting indexes.

• Adds 2-day aggregation and daily 7-day reconciliation windows.
• Moves daily/monthly aggregation to an hourly schedule while retaining 15-minute hourly metrics.
• Adds concurrent indexes for workflow execution queries.
• Introduces a WorkflowExecution-based organization prefilter that can omit organizations with activity only in other metric sources.

Confidence Score: 4/5

The PR should not merge until the active-organization prefilter includes organizations with qualifying activity in every metric source.

The new prefilter gates all aggregation on recent WorkflowExecution rows, so organizations with only PageUsage, Usage, or HITLQueue activity are skipped and retain stale or missing dashboard metrics.

Files Needing Attention: backend/dashboard_metrics/tasks.py

Important Files Changed

Filename	Overview
backend/dashboard_metrics/tasks.py	Implements tiered aggregation, narrower windows, monthly rollup, locking, and active-organization filtering; the prefilter can omit valid non-WorkflowExecution metric activity.
backend/dashboard_metrics/internal_views.py	Validates and forwards aggregation tier and source-window parameters across the internal scheduler boundary.
backend/dashboard_metrics/migrations/0005_add_reconciliation_task.py	Adds the daily seven-day reconciliation schedule for both Beat and PostgreSQL transports.
backend/dashboard_metrics/migrations/0006_split_aggregation_schedule.py	Splits hourly and daily/monthly aggregation schedules while preserving existing scheduler ownership.
backend/workflow_manager/file_execution/migrations/0007_wfe_status_created_idx.py	Adds a concurrent status-and-created-at index with separate migration state handling.
backend/workflow_manager/workflow_v2/migrations/0029_we_created_at_idx.py	Adds a concurrent created-at index with explicit validity and definition checks.
workers/scheduler/dashboard_metrics_tasks.py	Propagates tier and source-window scheduler arguments to the backend aggregation endpoint.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  S[Metric source tables] --> H[Hourly aggregation every 15 minutes]
  S --> D[Daily aggregation hourly]
  S --> R[Seven-day reconciliation daily]
  D --> DR[(EventMetricsDaily)]
  R --> DR
  DR --> M[Monthly rollup]
  H --> HR[(EventMetricsHourly)]
  M --> MR[(EventMetricsMonthly)]

Loading

Prompt To Fix All With AI

### Issue 1
backend/dashboard_metrics/tasks.py:631-634
**Prefilter drops valid metric activity**

When an organization has recent `PageUsage`, `Usage`, or `HITLQueue` activity but no `WorkflowExecution` created within the lookback, `_active_org_ids` excludes it from every metric query, causing its pages-processed, LLM, or HITL dashboard rows to remain stale or absent even after reconciliation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "Merge branch 'main' into UN-3883-Optimiz..." | Re-trigger Greptile}

[greptile-apps]

Prefilter drops valid metric activity

When an organization has recent PageUsage, Usage, or HITLQueue activity but no WorkflowExecution created within the lookback, _active_org_ids excludes it from every metric query, causing its pages-processed, LLM, or HITL dashboard rows to remain stale or absent even after reconciliation.

Knowledge Base Used: Restore dashboard metrics and navigation

Prompt To Fix With AI

This is a comment left during a code review.
Path: backend/dashboard_metrics/tasks.py
Line: 631-634

Comment:
**Prefilter drops valid metric activity**

When an organization has recent `PageUsage`, `Usage`, or `HITLQueue` activity but no `WorkflowExecution` created within the lookback, `_active_org_ids` excludes it from every metric query, causing its pages-processed, LLM, or HITL dashboard rows to remain stale or absent even after reconciliation.

**Knowledge Base Used:** [Restore dashboard metrics and navigation](https://app.greptile.com/zipstack/-/custom-context/knowledge-base/zipstack/unstract/-/reverts/incident-mitigation_1813-20260302-dashboard-metrics-sidebar-440ae49.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
✅	e2e-api-deployment	e2e	3	0	0	0	16.7
✅	e2e-coowners	e2e	1	0	0	0	1.8
✅	e2e-etl	e2e	1	0	0	0	20.9
✅	e2e-login	e2e	2	0	0	0	1.3
✅	e2e-prompt-studio	e2e	1	0	0	0	5.9
✅	e2e-smoke	e2e	2	0	0	0	1.5
✅	e2e-workflow	e2e	1	0	0	0	20.5
❌	frontend	unit	0	1	0	0	0.0
✅	integration-backend	integration	358	0	0	26	48.7
✅	integration-connectors	integration	1	0	0	7	8.0
✅	integration-workers	integration	157	0	0	1	51.0
❌	ui	e2e	0	1	0	0	0.0
❌	unit-backend	unit	1185	0	8	1	34.8
✅	unit-connectors	unit	63	0	0	0	9.7
✅	unit-core	unit	33	0	0	0	1.0
✅	unit-platform-service	unit	15	0	0	0	1.8
✅	unit-rig	unit	120	0	0	0	3.7
✅	unit-runner	unit	5	0	0	0	3.8
✅	unit-sdk1	unit	563	0	0	0	28.4
✅	unit-workers	unit	1376	0	0	1	114.2
	TOTAL		3887	2	8	36	373.9

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment