Skip to content

Make IndexedSet bulk removal linear via one-pass rebuild - #440

Open
LD-RW wants to merge 2 commits into
mahmoud:masterfrom
LD-RW:perf/indexedset-bulk-update
Open

Make IndexedSet bulk removal linear via one-pass rebuild#440
LD-RW wants to merge 2 commits into
mahmoud:masterfrom
LD-RW:perf/indexedset-bulk-update

Conversation

@LD-RW

@LD-RW LD-RW commented Jul 29, 2026

Copy link
Copy Markdown

Fixes #439

difference_update() and intersection_update() looped discard() per item.
Every 384 scattered removals, _cull() triggers a full O(n) _compact(),
making bulk removal of k items O(n·k/384) — quadratic when k scales with n.
See #439 for the flame graph and profile.

Change

  • Name the magic 384 threshold: _MAX_DEAD_INTERVALS (used in _cull and here)
  • Add private _bulk_discard(): rebuilds item_list/item_index_map in one
    O(n) pass and clears dead_indices
  • difference_update/intersection_update take the bulk path only when the
    removal count exceeds the threshold; small removals keep the existing
    incremental path unchanged
  • symmetric_difference_update deliberately untouched: its single-pass loop
    toggles order-dependently when other contains duplicates, which a bulk
    path cannot reproduce

Behavior

Identical: both methods already materialized the removal snapshot up front
(self.intersection(...) / self.difference(...)) before discarding, so
element order of survivors, indexing, and contents are unchanged. Verified with
randomized differential trials against the previous behavior (order,
__getitem__, and index() all checked) in addition to the test suite.

Numbers (Intel Core i7-12700H, Python 3.14.6)

difference_update removing every other element, best of 3:

n before after speedup
10,000 7.8 ms 1.7 ms 4.6x
20,000 24.6 ms 3.8 ms 6.5x
40,000 86.1 ms 7.7 ms 11x
80,000 317.3 ms 15.3 ms 21x
160,000 1,197.9 ms 31.2 ms 38x

Scaling is linear after the change (doubling n doubles time).

Tests

  • test_bulk_difference_update / test_bulk_intersection_update: exercise the
    new path (1000 elements > threshold), assert order, indexing, and post-op
    mutation behavior
  • test_small_difference_update_unchanged: pins the incremental path for
    small removals

LD-RW and others added 2 commits July 29, 2026 14:00
difference_update/intersection_update looped discard() per item; every
384 scattered removals that triggers a full O(n) _compact, making bulk
removal O(n*k/384). Rebuild once in O(n) when the removal count exceeds
the compaction threshold. Fixes #{{ISSUE_NUMBER}}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IndexedSet: difference_update/intersection_update are quadratic for large removals

2 participants