fix: table-growth snapshots (unbound SQL parameter) and slow brain-init revival - #39
Merged
Merged
Conversation
getTableStats built a query with NINE `?` placeholders — the two size
subtractions use two each — while the binding loop ran `for (i = 1; i <= 7)`.
Parameters 8 and 9 were never set, so Postgres rejected every call with
No value specified for parameter 8
That silently disabled table-growth snapshots for every table on every Postgres
connection: TableGrowthMonitoringService logs the failure per table and carries
on, so the scheduled job "succeeded" while capturing nothing. It shows up in the
backend log as a steady stream of
✗ Failed to capture snapshot for table: dba_batch_job_execution - No value
specified for parameter 8.
The query now binds one value and references it through a CTE, so the count
cannot drift again — counting placeholders by hand is precisely what failed here.
Verified against a live Postgres 18: the old form reports 9 parameters via
pg_prepared_statements, the new one reports 1 and returns correct sizes for
dba_batch_job_execution, the exact table from the log.
MySQL's getTableStats was checked and is unaffected (2 placeholders, 2 bound).
Tests: the existing getTableStats_returnsStats passed throughout the bug, because
a mocked PreparedStatement does not enforce that placeholders are bound. The new
test compares the two directly — it fails against the pre-fix provider with
"Wanted 9 times" while the other 10 tests still pass, which is exactly why this
reached production.
The window before db-scheduler reclaims an execution whose owner died is heartbeat-interval × missed-heartbeats-limit. At 5m × 6 that was thirty minutes. During it a dead brain-init stage is invisible rather than failed: the init-status endpoint keeps returning the last stage and percentage it reached, with completedAt null and errorMessage null, so a frozen run is indistinguishable from a slow one. smoke-test.sh waits 1200s (20m) — less than the revival window — so any run in which a stage died failed the smoke test even though initialization would have finished normally once revived. Observed exactly that: a stage stopped heartbeating, the smoke test timed out at 74%, and brain init then completed at 100% roughly 37 minutes after the scheduler revived it. 1m × 6 keeps the same six-missed-heartbeat tolerance and brings revival to about six minutes, comfortably inside the smoke-test window. Shortening the interval does not risk reclaiming healthy work: heartbeats are sent from the execution's own thread, so a long-running LLM call keeps heartbeating and is never mistaken for a dead owner. Only the heartbeat interval changes. application-test.properties overrides polling-interval and immediate-execution-enabled, not these keys, so tests are unaffected.
This was referenced Aug 9, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent defects found while running the self-host install flow end to end. Both were silent — each one degraded a feature while the surrounding job reported success.
1. Table-growth snapshots never captured on Postgres
PostgresIntrospectionProvider.getTableStatsbuilt a query with nine?placeholders (the two size subtractions use two each) while the binding loop ranfor (i = 1; i <= 7). Parameters 8 and 9 were never set, so every call failed with:TableGrowthMonitoringServicelogs this per table and carries on, so the scheduled job "succeeded" while capturing nothing — for every table, on every Postgres connection.The query now binds a single value and references it through a CTE, so the count cannot drift again.
Verified against a live Postgres 18: the old form reports 9 parameters via
pg_prepared_statements; the new one reports 1 and returns correct sizes fordba_batch_job_execution, the exact table from the log. MySQL'sgetTableStatswas checked and is unaffected (2 placeholders, 2 bound).2. Dead brain-init stages took 30 minutes to revive
The reclaim window is
heartbeat-interval × missed-heartbeats-limit=5m × 6= 30 minutes.During it a dead stage is invisible rather than failed:
init-statuskeeps returning the last stage and percentage, withcompletedAtanderrorMessageboth null — a frozen run looks exactly like a slow one.smoke-test.shwaits 1200s (20m), less than the revival window, so any run where a stage died failed the smoke test even though initialization would have completed normally.Observed exactly that: a stage stopped heartbeating, the smoke test timed out at 74%, and brain init then reached 100% about 37 minutes after the scheduler revived it.
1m × 6keeps the same six-missed-heartbeat tolerance and brings revival to ~6 minutes, inside the smoke-test window. Shortening the interval does not risk reclaiming healthy work — heartbeats come from the execution's own thread, so a long LLM call keeps heartbeating.Testing
PostgresIntrospectionProviderTest— 11/11 pass.Wanted 9 times, while the other 10 tests still pass — which is why this reached production:getTableStats_returnsStatspassed throughout, because a mockedPreparedStatementdoes not enforce that placeholders are bound.application-test.propertiesoverridespolling-intervalandimmediate-execution-enabled, not the heartbeat keys, so tests are unaffected by change 2.Not addressed here
TableGrowthMonitoringService.captureSnapshotis@Transactionalandprivate. Spring cannot proxy a private method, and both call sites reach it by self-invocation, so the annotation is inert — while its Javadoc claims it "ensures all repository operations use a single database connection, reducing connection pool pressure from 4-5 connections per table to just 1." That claim does not hold today. Left out of this PR because fixing it changes transaction boundaries and deserves its own review.