Buffers & Bytes & Zerocopy

[somethingelseentirely]

Hey,
I saw vortex on HN and figured I'd drop by for a chat.
I'm operating in a similar space (building triple stores), and similarly hit the lack of Bytes extensibility when implementing zero-copy mmap-ed reads.
After surveying the space I decided to resurrect minibytes as anybytes.
It uses the same trait based approach that minibytes and ownedbytes (a crate written and mainly used by the Quickwit fulltext-search project) use, but has better Zerocopy compatibility. (it's usable for experimentation but I want to do a full model check with Kani before it hits 1.0)

I figured it might be of interest for you guys as you too seem to operate in a mostly read-only regime, and I discovered that it can be surprisingly easy to get safe zerocopy bytes <-> types transmutes in that space.

Epsilon-Serde might also be of interest to you, and it's somewhat hard to find if you're not operating in the succinct datastructure space. The basic idea is that a lot of read-only data-structures can be conveniently represented as arrays (cough columns) that are mmap-ed, with only a small allocation and dynamic deserialization of header information.

Hope this is of some relevance.

[gatesn]

Hi @somethingelseentirely, thanks for dropping by. We'll definitely look into anybytes. As you might have seen, for now we have a https://github.com/spiraldb/vortex/tree/develop/vortex-buffer crate that wraps an enum of Tokio Bytes and Arrow Bytes, which isn't ideal, but at least gives us a place to swap these things out.

A lot of the problems we faced were on requesting / guaranteeing alignment. For now we just lean on our existing primitive types, including a u128, in order to force it at allocation time. But it's a little bit more fragile than I'd like.

The other feature we use quite often is the ability to optimistically convert immutable bytes back into mutable bytes, e.g. if we hold the only strong reference.

---

In terms of serde formats, while we hope to never have to implement or maintain a non-Rust Vortex, we decided to go with flatbuffers as it was more portable than some of the alternatives, and seemed to fit our use-case more closely than e.g. cap'n'proto.

But @robert3005 - we could steal the lifetime erasure trick to provide typed accessors to flat buffers, instead of passing around byte slices and positions? https://github.com/triblespace/anybytes/blob/0d384ca2a17b7fb13ee24ec03e513ed0de974e33/src/bytes.rs#L158

Thank you very much for the links though, it's given us some things to think out.

[robert3005]

Something still needs to hold the bytes, all this lets you do is swap the position to deserialized flatbuffer

[somethingelseentirely]

Your alignment requirements are an interesting issue that I'll have to think about. I haven't encountered such issues yet, because the bytes were either aligned by construction (a Vec<T> derived Bytes via zerocopy has the alignment of T), or because Packed* perform a runtime check for alignment through zerocopy (which I mostly use for mmap bytes, but those have such large alignment that it's generally not relevant).

Having an abstraction that allows you to go back to mutable bytes is a hard problem indeed, and I'm not sure if it can be done in a way that is truly agnostic to the underlying storage, e.g. mmap read only bytes behave pretty much exactly like other in-memory bytes, but writing them reliably is a bit of witchcraft. 😅

Looking forward to see how Vortex will evolve!