Make IndexedSet bulk removal linear via one-pass rebuild - #440
Open
LD-RW wants to merge 2 commits into
Open
Conversation
difference_update/intersection_update looped discard() per item; every
384 scattered removals that triggers a full O(n) _compact, making bulk
removal O(n*k/384). Rebuild once in O(n) when the removal count exceeds
the compaction threshold. Fixes #{{ISSUE_NUMBER}}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #439
difference_update()andintersection_update()loopeddiscard()per item.Every 384 scattered removals,
_cull()triggers a full O(n)_compact(),making bulk removal of k items O(n·k/384) — quadratic when k scales with n.
See #439 for the flame graph and profile.
Change
384threshold:_MAX_DEAD_INTERVALS(used in_culland here)_bulk_discard(): rebuildsitem_list/item_index_mapin oneO(n) pass and clears
dead_indicesdifference_update/intersection_updatetake the bulk path only when theremoval count exceeds the threshold; small removals keep the existing
incremental path unchanged
symmetric_difference_updatedeliberately untouched: its single-pass looptoggles order-dependently when
othercontains duplicates, which a bulkpath cannot reproduce
Behavior
Identical: both methods already materialized the removal snapshot up front
(
self.intersection(...)/self.difference(...)) before discarding, soelement order of survivors, indexing, and contents are unchanged. Verified with
randomized differential trials against the previous behavior (order,
__getitem__, andindex()all checked) in addition to the test suite.Numbers (Intel Core i7-12700H, Python 3.14.6)
difference_updateremoving every other element, best of 3:Scaling is linear after the change (doubling n doubles time).
Tests
test_bulk_difference_update/test_bulk_intersection_update: exercise thenew path (1000 elements > threshold), assert order, indexing, and post-op
mutation behavior
test_small_difference_update_unchanged: pins the incremental path forsmall removals