Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vortex should support zero-copy roaring int and roaring bool array construction #1075

Open
danking opened this issue Oct 17, 2024 · 1 comment

Comments

@danking
Copy link
Member

danking commented Oct 17, 2024

Both RoaringIntArray and RoaringBoolArray use Bitmap::deserialize which copies the source bytes. There exists BitmapView::deserialize but it is unsafe. We either should verify the correctness of the source bytes and use the unsafe method or we should find another way to deserialize a roaring array without copying.

In my experience working with the PBI datasets, for example CMSprovider, copying roaring arrays can be ~10% of total runtime when decompressing.

danking added a commit that referenced this issue Oct 17, 2024
Until the roaring compressors can deserialize quickly (i.e. without copies, see #1075),
we disable them to improve the decompression throughput of Vortex.
danking added a commit that referenced this issue Oct 17, 2024
Until the roaring compressors can deserialize quickly (i.e. without copies, see #1075),
we disable them to improve the decompression throughput of Vortex.
@robert3005
Copy link
Member

There should be array validator that you could implement for an array then it would happen once on construction. We minimally try to deserialize metadata but that doesn't happen all the time and doesn't cover the case here. Then we could use the View types

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants