feat: signal deduplication + follower-count influence weighting - #512
Open
essentialbit wants to merge 10 commits into
Open
feat: signal deduplication + follower-count influence weighting#512essentialbit wants to merge 10 commits into
essentialbit wants to merge 10 commits into
Conversation
… signals MISSION.md's L2 checklist has named "Signal deduplication and influence weighting (high-follower accounts)" as an open item since project inception, and CLAUDE.md's own step-3 priority order ranks signal quality above dashboard/coverage/performance work -- but no dedup or weighting logic existed anywhere in the ingestion path. twitter_client.py now fetches each author's public_metrics (follower count) and computes a bounded [1.0, 3.0] log-scale influence_weight per signal, and fetch_signals() dedupes near-identical retweets/quote-tweets of the same viral text within a collection cycle (URL-stripped, whitespace-normalized signature) before they're persisted as N independent signals. trend_detector's compute_sentiment_stats() now computes a weight-adjusted average instead of a flat mean, falling back to neutral weight 1.0 for any signal (non-Twitter sources included) that carries no influence_weight in its metadata -- output shape and non-Twitter behavior are both unchanged.
essentialbit
pushed a commit
that referenced
this pull request
Jul 25, 2026
L3-L6 sections were already refreshed on this branch; the top-of-doc L2 section (lines 33-46) was the one part still using original placeholder bullets despite the L2 Priority queue section further down already marking all 9 items done. All 10 L2 items verified against real modules on main; 9 are merged, 1 (signal dedup/influence weighting) is code-complete on open PR #512 pending merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
twitter_client.py:search_recent()now requestspublic_metricson the author expansion (follower count) and computes a bounded[1.0, 3.0]log-scaleinfluence_weightper tweet (0 followers -> 1.0, ~1000 -> ~2.0, capped at 3.0 so no single mega-account can swamp an aggregate).fetch_signals()now dedupes near-identical retweets/quote-tweets of the same viral text within a collection cycle (URL-stripped, whitespace-normalized signature) before they're persisted as N independent signals, and storesfollowers/influence_weightin the existingmetadataJSON blob (no schema migration).trend_detector.py:compute_sentiment_stats()now computes a weight-adjusted average instead of a flat mean, falling back to a neutral weight of 1.0 for any signal (non-Twitter sources included) that carries noinfluence_weight-- output shape and non-Twitter/no-metadata behavior are both unchanged.Test plan
data/sentinel.db): influence-weight bounds/monotonicity, dedup-signature normalization, an end-to-end mockedfetch_signals()run confirming a near-duplicate retweet is dropped and only 2 of 3 synthetic tweets persist, weighted-vs-unweighted average divergence, and backward-compatible neutral-weight fallback for metadata-less/non-Twitter signals -- all passed.python3 -c "from main import *"import smoke test -- passes.tests/directory exists in this repo (documented convention); no persisted test regression here.🤖 Generated with headless FredAI sensor cycle (Claude Code)