duckdb: propagate per-file "partition" stats - #9639
Conversation
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_avx2[16384] |
17.7 µs | 21.3 µs | -16.81% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 13.5 µs | +50.76% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.3 µs | 17.6 µs | +20.72% |
| ⚡ | Simulation | take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000] |
245 µs | 207 µs | +18.39% |
| ⚡ | WallTime | words_gather_dispatch_avx512[1024] |
9 ns | 8 ns | +12.5% |
| ⚡ | WallTime | mul_u32_nonnull_avx512 |
6.3 µs | 5.7 µs | +10.4% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing myrrc/duckdb-partition-stats (8b3244b) with develop (ee1ac25)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
Pull request was converted to draft
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.010x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.978x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.027x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.978x ➖, 1↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 52↑ 48↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 3↑ 3↓)
duckdb / parquet / ns (0.984x ➖, 8↑ 2↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.134x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.913x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.694x ✅, 5↑ 1↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.994x ➖, 3↑ 2↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.909x ➖, 2↑ 2↓)
duckdb / parquet / ns (0.978x ➖, 2↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.937x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.957x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.961x ➖, 0↑ 0↓)
|
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
This couldn't be done previously since we didn't have ordering of files
for threads. Since migration to MultiFileReader, we now have well-defined
batch indices.
Propagate per-file partition stats to duckdb. MultiFileSession already caches
footers by file path, so we can use this information. This change makes
Clickbench Q0 and Q6 constant (supplied from metadata) from second iteration.
Resolves: #3393