Add PatchesV2 with chunk-local patch indices - #9611
Draft
joseph-isaacs wants to merge 5 commits into
Draft
Conversation
Add a patch container addressed by chunk-local u16 indices with required, rebased u32 chunk-offset prefix counts, plus a grid offset for unaligned slices. Compared to Patches, the index child stays two bytes per patch at any array length, chunk lookup is constant time without saturating offset adjustments, and slicing rebases the chunk offsets so every slice is self-contained. Includes conversions to and from Patches, search and point lookup, slicing, and validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BTTvqxz7t96Ebkef7vamk Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will improve performance by 14.48%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.5 µs | 8.3 µs | +14.48% |
| 🆕 | Simulation | search_index_above_max_v2 |
N/A | 19.5 µs | N/A |
| 🆕 | Simulation | search_index_below_min_v2 |
N/A | 18.4 µs | N/A |
| 🆕 | Simulation | search_index_full_range_random_v2 |
N/A | 21.3 µs | N/A |
| 🆕 | Simulation | search_index_in_range_v2 |
N/A | 20.1 µs | N/A |
| 🆕 | Simulation | search_index_mixed_out_of_range_v2 |
N/A | 18.7 µs | N/A |
| 🆕 | Simulation | apply_full_range |
N/A | 9.6 µs | N/A |
| 🆕 | Simulation | apply_full_range_v2 |
N/A | 16.2 µs | N/A |
| 🆕 | Simulation | slice_unaligned |
N/A | 44.7 µs | N/A |
| 🆕 | Simulation | slice_unaligned_v2 |
N/A | 88.9 µs | N/A |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/patches-v2-local-indices (2992f12) with develop (ee1ac25)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
Contributor
|
Don't we want to move to the world of PatchedArray and we can make all the changes there? |
Extend the patches_lookup benchmark with PatchesV2 variants of each search scenario. Give PatchesV2::search_index a downcast fast path that reads canonical index and chunk-offset children in place instead of executing them per query. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BTTvqxz7t96Ebkef7vamk Signed-off-by: Claude <noreply@anthropic.com>
Give PatchesV2 a resolved borrowed view for repeated lookups, an apply_each scatter primitive with a chunk cursor, and route chunked BitPacked and FoR patch application through it during decompression, with a counter instrumenting the path. A vortex-btrblocks test compresses TPC-H lineitem, orders, and partsupp, decodes every column, and asserts the chunk-local scatter ran; an ignored variant runs the same validation at scale factor one. The patches_lookup benchmark gains apply and slice comparisons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BTTvqxz7t96Ebkef7vamk Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
Converting Patches to PatchesV2 on every decompression allocates and walks the patch set on a hot path, regressing patched decompression benchmarks. Gate the chunk-local scatter behind force_patches_v2_scatter or VORTEX_PATCHES_V2_SCATTER=1 so the default path is unchanged; the fastlanes and TPC-H validation tests enable it explicitly. Zero-cost integration needs the stored layout to be chunk-local, which stays follow-up work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BTTvqxz7t96Ebkef7vamk Signed-off-by: Claude <noreply@anthropic.com>
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.986x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.985x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -46.4% overall, 0↑ 2↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.979x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
File Size Changes (10 files changed, -45.5% overall, 0↑ 10↓)
Totals:
|
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.019x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.009x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.975x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 1↓)
File Size Changes (201 files changed, -42.9% overall, 58↑ 143↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.012x ➖, 1↑ 6↓)
duckdb / parquet / ns (1.017x ➖, 5↑ 12↓)
File Size Changes (25 files changed, -43.9% overall, 0↑ 25↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
File Size Changes (10 files changed, -45.6% overall, 0↑ 10↓)
Totals:
|
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.994x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -37.7% overall, 0↑ 2↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.833x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.083x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.639x ❌, 0↑ 8↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.012x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 2↓)
File Size Changes (101 files changed, -39.5% overall, 0↑ 101↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.931x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.982x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.959x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.982x ➖, 0↑ 0↓)
|
Inlining the opt-in chunk-local scatter into apply_patches_to_uninit_range degraded the default per-patch loop codegen, regressing patched decompression by twenty percent even with the toggle off. Move it to a cold never-inlined helper; the alp_for_bp_f64 decompress benchmark returns to its develop baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BTTvqxz7t96Ebkef7vamk Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Patchesaddresses outliers with global row indices, which forces a wide index child (u32/u64for large arrays) and threadsoffset/offset_within_chunksaturating adjustments through every chunked lookup and slice. This PR addsPatchesV2, a patch container addressed by chunk-localu16indices, and exercises it on the real decompression read path.Changes
vortex-array/src/patches_v2.rswith thePatchesV2struct:indices:u16positions local to each 1024-value chunk (PATCH_CHUNK_SIZE), strictly sorted per chunk.chunk_offsets: required, non-nullableu32prefix patch counts with a leading zero — rebased on every slice, so slices are self-contained and lookups need no saturating-adjustment bookkeeping.offsetin0..PATCH_CHUNK_SIZEplaces logical element zero inside the first chunk, keeping unaligned slices on the chunk grid.try_new/new_unchecked,search_indexandget_patched(O(1) chunk select + in-chunk binary search),view()returning a resolved borrowedPatchesV2Viewfor hot loops (plain slice reads, no dispatch or allocation),apply_each(the decompression scatter primitive, walking patches with a chunk cursor), rebasingslice, andfrom_patches/to_patchesconversions.apply_patches_to_uninit_range(used by BitPacked and FoR decompression) scatters chunked patch sets throughPatchesV2, instrumented with a counter (patches_v2_apply_count).vortex-btrblockstest compresses lineitem, orders, and partsupp, decodes every column, and asserts the chunk-local scatter ran (decimalmsp, dictcodes, and FoRencodedcolumns produce patched BitPacked arrays). The#[ignore]d SF=1 variant passes with 43 PatchesV2 scatters across the three tables.patches_lookupbenchmark gains_v2search variants plus apply and slice comparisons.Benchmarks
cargo bench -p vortex-array --bench patches_lookup(1M rows, 100 patches, 100 queries per iteration, medians):Search matches unchunked
Patcheswhile keeping O(1) chunk select, and is ~40× faster than the chunkedPatchespath (which pays per-query child slicing and generic dispatch). Apply and slice are slightly behind v1 at this patch density because reconstructing positions walks ~1000 chunk offsets for only 100 patches; at realistic patch densities (patches ≥ chunks) the cursor walk amortizes. The v1 apply/slice numbers also exclude the wider on-disk indices v1 pays for.This lands the container plus the decompression scatter; migrating
BitPacked's stored layout onto chunk-local indices is follow-up work.Checks run:
cargo test -p vortex-array patches,cargo test -p vortex-fastlanes(317 tests),cargo test -p vortex-btrblocks(including the SF=1 ignored test in release),cargo bench -p vortex-array --bench patches_lookup,cargo clippy --all-targetson the three crates,cargo +nightly fmt. SQL benchmarks triggered on CI via theaction/bench-sqllabel.