Skip to content

fix(bench): include file open in random-access reopen timings - #9631

Draft
lwwmanning wants to merge 1 commit into
developfrom
wm/random-access-reopen-timing
Draft

fix(bench): include file open in random-access reopen timings#9631
lwwmanning wants to merge 1 commit into
developfrom
wm/random-access-reopen-timing

Conversation

@lwwmanning

Copy link
Copy Markdown
Contributor

Summary

--open-mode reopen creates each accessor outside the recorded timing interval. Its -footer measurements therefore exclude file-open and metadata parsing costs.

Changes

Create the accessor inside each timed reopen iteration. Cached mode still opens once and reuses its accessor.

🤖 Generated with Claude Code

Signed-off-by: Will Manning <will@willmanning.io>
@codspeed-hq

codspeed-hq Bot commented Aug 25, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 21.99%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
✅ 2108 untouched benchmarks
⏩ 106 skipped benchmarks1
🗄️ 4 archived benchmarks run2

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime arrow_checked_add_u32_neon[16384] 20.4 µs 12.6 µs +62.12%
Simulation compact[(2048, 90)] 1.5 µs 1.4 µs +11.89%
Simulation compress_fsst[(500, 64, 4)] 488.1 µs 441.6 µs +10.52%
Simulation compress_fsst[(500, 64, 8)] 524.9 µs 475.1 µs +10.48%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing wm/random-access-reopen-timing (e003545) with develop (ee1ac25)

Open in CodSpeed

Footnotes

  1. 106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. 4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them.

@joseph-isaacs joseph-isaacs added the action/bench-random-access Run only the random-access benchmark on this PR label Aug 26, 2026
@github-actions github-actions Bot removed the action/bench-random-access Run only the random-access benchmark on this PR label Aug 26, 2026
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done e003545 random-access-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Random Access 📖

Commits: PR e0035458 vs base ee1ac252
Verdict: Likely regression (medium confidence)
Attributed Vortex impact: +27.8%
Engines: random-access Likely regression (+27.8%, medium confidence)
Vortex (geomean): hot 1.281x ❌
Parquet (geomean): hot 1.002x ➖
Shifts: Parquet (control) +0.2% · Median polish +6.5%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
  • Hot vs cold: Every measurement is run several times. The first run is reported as the cold run, and the median of the runs after it is reported as the hot run. The verdict and significance use hot runs; each target's geomean reports hot and cold beside each other where the individual runs were recorded and hot alone where they were not, the cold column shows first-run cost per row, and hot/cold is how much of each run the warm path saves. Rows whose results predate per-run reporting show only one value, taken from the value the runner reported.
  • Table cells: Each cell reads PR / base / %diff. The hot and cold columns mark a change as 🔴 slower or 🟢 faster once it clears this suite's threshold; anything smaller is noise here and is left unmarked, as is hot/cold, because a shift in the warm-up ratio is not a win or a loss by itself.

random-access / vortex-file-compressed / ns (1.281x ❌, 0↑ 10↓)
name ns (PR / base / %diff)
random-access/feature-vectors/correlated/vortex-tokio-local-disk 2190406 / 370356 / +491.4% 🔴
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer 1666957 / 607467 / +174.4% 🔴
random-access/feature-vectors/uniform/vortex-tokio-local-disk 2628639 / 2639181 / -0.4%
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer 4499242 / 3065319 / +46.8% 🔴
random-access/nested-lists/correlated/vortex-tokio-local-disk 286945 / 289739 / -1.0%
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer 446431 / 367132 / +21.6% 🔴
random-access/nested-lists/uniform/vortex-tokio-local-disk 1648816 / 1662395 / -0.8%
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer 1961579 / 1751013 / +12.0% 🔴
random-access/nested-structs/correlated/vortex-tokio-local-disk 340586 / 327042 / +4.1%
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer 602612 / 464998 / +29.6% 🔴
random-access/nested-structs/uniform/vortex-tokio-local-disk 1019607 / 1013230 / +0.6%
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer 1394558 / 1198702 / +16.3% 🔴
random-access/taxi/correlated/vortex-tokio-local-disk 926984 / 903817 / +2.6%
random-access/taxi/correlated/vortex-tokio-local-disk-footer 1768570 / 1420918 / +24.5% 🔴
random-access/taxi/uniform/vortex-tokio-local-disk 4070628 / 4105773 / -0.9%
random-access/taxi/uniform/vortex-tokio-local-disk-footer 5419880 / 4838072 / +12.0% 🔴
random-access/vortex-tokio-local-disk 580864 / 596405 / -2.6%
random-access/vortex-tokio-local-disk-footer 1251374 / 1000368 / +25.1% 🔴
random-access / parquet / ns (1.002x ➖, 0↑ 0↓)
name ns (PR / base / %diff)
random-access/feature-vectors/correlated/parquet-tokio-local-disk 8894414495 / 8987163898 / -1.0%
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer 8916668735 / 9053539482 / -1.5%
random-access/feature-vectors/uniform/parquet-tokio-local-disk 9099600983 / 9071046804 / +0.3%
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer 9004593682 / 8939656424 / +0.7%
random-access/nested-lists/correlated/parquet-tokio-local-disk 138083440 / 136621663 / +1.1%
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer 136158954 / 135072389 / +0.8%
random-access/nested-lists/uniform/parquet-tokio-local-disk 135808436 / 134854836 / +0.7%
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer 138413325 / 135023641 / +2.5%
random-access/nested-structs/correlated/parquet-tokio-local-disk 20220903 / 20172969 / +0.2%
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer 19041658 / 19053294 / -0.1%
random-access/nested-structs/uniform/parquet-tokio-local-disk 18919528 / 18952771 / -0.2%
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer 19416977 / 18865365 / +2.9%
random-access/parquet-tokio-local-disk 180166287 / 180877773 / -0.4%
random-access/parquet-tokio-local-disk-footer 180253827 / 181143674 / -0.5%
random-access/taxi/correlated/parquet-tokio-local-disk 270558880 / 271655515 / -0.4%
random-access/taxi/correlated/parquet-tokio-local-disk-footer 270499845 / 271655437 / -0.4%
random-access/taxi/uniform/parquet-tokio-local-disk 285717583 / 286593764 / -0.3%
random-access/taxi/uniform/parquet-tokio-local-disk-footer 288532502 / 288970163 / -0.2%
random-access / lance / ns (1.104x ❌, 0↑ 8↓)
name ns (PR / base / %diff)
random-access/feature-vectors/correlated/lance-tokio-local-disk 354945 / 348020 / +2.0%
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer 1357406 / 1105640 / +22.8% 🔴
random-access/feature-vectors/uniform/lance-tokio-local-disk 1069366 / 1089784 / -1.9%
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer 2134082 / 1835935 / +16.2% 🔴
random-access/lance-tokio-local-disk 639026 / 658389 / -2.9%
random-access/lance-tokio-local-disk-footer 1655927 / 1333601 / +24.2% 🔴
random-access/nested-lists/correlated/lance-tokio-local-disk 184472 / 186637 / -1.2%
random-access/nested-lists/correlated/lance-tokio-local-disk-footer 822950 / 575880 / +42.9% 🔴
random-access/nested-lists/uniform/lance-tokio-local-disk 896442 / 915547 / -2.1%
random-access/nested-lists/uniform/lance-tokio-local-disk-footer 1596435 / 1304329 / +22.4% 🔴
random-access/nested-structs/correlated/lance-tokio-local-disk 309246 / 316394 / -2.3%
random-access/nested-structs/correlated/lance-tokio-local-disk-footer 701039 / 454993 / +54.1% 🔴
random-access/nested-structs/uniform/lance-tokio-local-disk 2397452 / 2420329 / -0.9%
random-access/nested-structs/uniform/lance-tokio-local-disk-footer 2799730 / 2536278 / +10.4% 🔴
random-access/taxi/correlated/lance-tokio-local-disk 950641 / 939482 / +1.2%
random-access/taxi/correlated/lance-tokio-local-disk-footer 2171860 / 1846971 / +17.6% 🔴
random-access/taxi/uniform/lance-tokio-local-disk 10082739 / 9957337 / +1.3%
random-access/taxi/uniform/lance-tokio-local-disk-footer 10878953 / 10663486 / +2.0%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants