Scope extractor_runs queries to candidate filings, not corpus

[sroussey]

Summary

This change refactors how the system queries successful extractor runs to dramatically reduce memory usage during large-scale processing. Instead of loading all successful runs for an extractor into memory at once, queries are now scoped to only the candidate filings being examined, with results chunked to respect SQLite's parameter binding limits.

Key Changes

• ExtractorRunRepo: Renamed successfulRunKeys() to successfulRunKeysForFilings() and refactored to:

  • Accept the candidate filings as input and scope queries to only their CIKs
  • Chunk CIK lists by 900 (respecting SQLite's SQLITE_MAX_VARIABLE_NUMBER limit)
  • Filter results to only include keys present in the candidate set
  • Eliminates the need to hold corpus-wide Sets in memory

• ComputeFormsWorklistTask: Updated to:

  • Call successfulRunKeysForFilings() once per page with only the eligible filings
  • Extract cheap in-memory tests (shard, CIK allow-list, filing date) into passesCheapTests() method
  • Apply cheap tests before the database lookup to minimize query scope
  • Removed the form-scoped successfulKeys cache that was held for the entire form scan

• backfillDescriptors.ts: Refactored filing selection to:

  • Introduce pageFilingsOfForm() generator for keyset-resume pagination
  • Page through filings with BACKFILL_PAGE_SIZE = 10_000 to bound memory
  • Apply paging to both selectFilingsByForms() and spacTrigger8KSelector()
  • Reduces peak memory from hundreds of MB to ~5 MB per page

• ProcessSpacTimelineTask: Updated loadSuccessfulKeys() to:

  • Accept the timeline filings and scope queries to only that issuer's filings
  • Eliminates corpus-wide queries that were repeated per CIK in sec sync spacs

• Constants.ts: Added SecSqliteCacheMb configuration:

  • Replaces hardcoded cache_size = 1000000 (which was ~4 GB) with configurable 256 MB default
  • Clamped to 2..4096 MB range via SEC_SQLITE_CACHE_MB env var
  • Reduces RSS growth during long form sweeps from +31 MB to +3 MB

• db.ts: Updated SQLite pragma to use negative cache_size (KiB form) with the new constant

Notable Implementation Details

• The chunking strategy in successfulRunKeysForFilings() is necessary because SQLite has no two-column IN operator; querying by CIK returns all accessions for those CIKs, so results must be filtered against the candidate set to prevent a high-volume filer from inflating the Set beyond the page size.

• Keyset-resume pagination in pageFilingsOfForm() uses two queries per page: one for remaining accessions of the last CIK, then one for later CIKs. This prevents stalling on serial filers with more filings than the page size.

• The cheap tests (shard, CIK allow-list, filing date) are applied before the extractor_runs lookup to minimize query scope—under --shard 1/6 this reduces the lookup to ~1/6 of the page.

• All changes maintain backward compatibility; the refactoring is purely an optimization that changes memory characteristics without altering query results.

Testing

Added three new test cases to ExtractorRunRepo.test.ts:

• Verifies that only candidate CIKs are queried (not the corpus)
• Verifies that a CIK's other accessions are filtered out
• Verifies that chunking works correctly past the SQLite bind parameter limit (1000 CIKs)

https://claude.ai/code/session_01CsYGzG99SnoxeQbdAqijBS

[Copilot]

Pull request overview

This PR refactors how successful extractor_runs are queried during large sweeps so lookups are scoped to the current candidate filings/page instead of materializing corpus-wide sets, significantly reducing peak memory usage during processing.

Changes:

• Refactors ExtractorRunRepo to query successful runs only for the candidate filings’ CIKs (chunked to stay under SQLite bind limits) and filter results back to the candidate set.
• Updates forms sweep and SPAC timeline processing to apply cheap in-memory eligibility checks first, then perform one scoped extractor_runs lookup per page/issuer.
• Introduces a configurable SQLite page-cache ceiling (SEC_SQLITE_CACHE_MB, default 256 MB) and applies it via a negative PRAGMA cache_size (KiB units).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/storage/versioning/ExtractorRunRepo.ts	Replaces corpus-wide successful-run materialization with candidate-scoped, chunked lookups filtered to the candidate set.
src/storage/versioning/ExtractorRunRepo.test.ts	Adds tests validating scoping to candidate CIKs, filtering out non-candidate accessions, and chunking beyond SQLite bind limits.
src/task/forms/ComputeFormsWorklistTask.ts	Moves to per-page successful-run lookups after cheap eligibility filters; removes long-lived per-form successful-key cache.
src/task/forms/backfillDescriptors.ts	Adds paged form scanning (pageFilingsOfForm) to avoid loading entire form corpuses into memory during backfill candidate selection.
src/task/spac/ProcessSpacTimelineTask.ts	Scopes successful-run key loading to the current issuer’s timeline filings and routed extractor ids.
src/config/Constants.ts	Adds SecSqliteCacheMb with env override + clamping to control SQLite pager cache growth.
src/util/db.ts	Applies the new cache ceiling using negative PRAGMA cache_size (KiB form).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.