You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Both RoaringIntArray and RoaringBoolArray use Bitmap::deserialize which copies the source bytes. There exists BitmapView::deserialize but it is unsafe. We either should verify the correctness of the source bytes and use the unsafe method or we should find another way to deserialize a roaring array without copying.
In my experience working with the PBI datasets, for example CMSprovider, copying roaring arrays can be ~10% of total runtime when decompressing.
The text was updated successfully, but these errors were encountered:
Until the roaring compressors can deserialize quickly (i.e. without copies, see #1075),
we disable them to improve the decompression throughput of Vortex.
Until the roaring compressors can deserialize quickly (i.e. without copies, see #1075),
we disable them to improve the decompression throughput of Vortex.
There should be array validator that you could implement for an array then it would happen once on construction. We minimally try to deserialize metadata but that doesn't happen all the time and doesn't cover the case here. Then we could use the View types
Both
RoaringIntArray
andRoaringBoolArray
useBitmap::deserialize
which copies the source bytes. There existsBitmapView::deserialize
but it is unsafe. We either should verify the correctness of the source bytes and use the unsafe method or we should find another way to deserialize a roaring array without copying.In my experience working with the PBI datasets, for example CMSprovider, copying roaring arrays can be ~10% of total runtime when decompressing.
The text was updated successfully, but these errors were encountered: