Skip to content

fix(preprocess): skip article fetches for incomplete discovery - #151

Open
pengpengyi92 wants to merge 1 commit into
LLMQuant:masterfrom
pengpengyi92:codex/fix-incomplete-news-fetch-144
Open

fix(preprocess): skip article fetches for incomplete discovery#151
pengpengyi92 wants to merge 1 commit into
LLMQuant:masterfrom
pengpengyi92:codex/fix-incomplete-news-fetch-144

Conversation

@pengpengyi92

@pengpengyi92 pengpengyi92 commented Aug 22, 2026

Copy link
Copy Markdown

Summary

  • Stop PR Newswire collection before article-body fetching when listing discovery is incomplete.
  • Preserve discovery failures and the observed row count while returning no partial documents, so completeness-preserving retries do not download the same parent-window articles first.
  • Document the incomplete-window contract and add a focused regression test proving the article collector is not invoked.

This PR deliberately addresses only the redundant-fetch defect. The independent proposal to expose public article-fetch concurrency remains out of scope so that any configuration API can be reviewed and benchmarked separately.

Related Issue

Part of #144

Verification

  • bash scripts/verify.sh in the project's Linux test environment: 486 passed, 86.46% coverage, import contracts and static checks passed.
  • python -m pytest tests/preprocess/test_pr_newswire.py tests/flows/test_news.py tests/configs/test_news.py -q -o addopts=: 19 passed.
  • python scripts/verify_news_e2e.py: PR Newswire discovery passed (332 observed rows across 4 pages) and ticker recovery passed (8/8 expected tickers); the independent RSS smoke check failed because the configured public PR Newswire RSS URL currently returns HTTP 404.
  • A live 14-day collection probe reached the 2,000-row discovery cap and returned complete=False, documents=0, and the discovery failure evidence without collecting partial article bodies.

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • bash scripts/verify.sh passes.
  • Every applicable live-network component smoke test passes, or this PR states why none applies.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant