Fetch only what you need. Store each unique chunk once.
SparseIO is infrastructure and an extensible Rust library for coordinating sparse, out-of-order ranged fetches to materialize large-object content-addressable storage (CAS).
Large objects are often consumed a few ranges at a time: a tensor from a model, a row group from a dataset, several blocks from a backup, or a segment from a media file. Fetching the entire object before serving the first useful byte wastes time, bandwidth, and storage. SparseIO materializes an object incrementally instead. A requested range is fetched from its upstream source, split into stable chunks, and stored by content hash so future reads can reuse it.
| Whole-object caching | SparseIO |
|---|---|
| Downloads every byte on the first miss | Fetches only the ranges callers request |
| Stores repeated data once per object | Deduplicates equal chunks by content hash |
| Can duplicate work during concurrent misses | Coalesces in-flight requests for the same chunk |
| Couples the cache to a source or runtime | Uses pluggable, executor-neutral backend traits |
This is especially useful for:
- AI/ML models and datasets where only selected tensors or shards are needed.
- Database backups, VM images, and archives explored without a full restore.
- Columnar data, logs, and scientific data read non-sequentially.
- Media and other large remote objects served through byte-range requests.
For each range read, SparseIO:
- Normalizes the requested range into fixed-size chunks.
- Looks up each chunk in the metadata index and local cache.
- Fetches missing chunks from the registered upstream
Reader. - Coalesces concurrent misses so only one upstream fetch does the work.
- Hashes and writes new chunks into the CAS through the configured
Writer. - Returns the requested bytes while the object becomes incrementally available.
Because chunks are addressed by their content, identical regions can be shared across objects and versions. A fine-tuned model, incremental database backup, or revised disk image only needs storage for the chunks that actually changed.
SparseIO is designed to work at two levels:
- Embedded library: compose storage systems directly in Rust using small,
object-safe
Reader,Writer, andMetadatatraits. The core remains independent of Tokio or any other specific async executor. - Deployable infrastructure: expose sparse materialization to applications and non-Rust clients through service interfaces.
Planned infrastructure includes:
- A Redis/RESP interface for accessing SparseIO from existing clients and tooling.
- In-memory peers and trackers inspired by Meta's Owl architecture for high-fanout, peer-assisted chunk distribution. Peers cache and transfer chunks; trackers coordinate where peers fetch them and maintain a view of distribution state.
The backend contracts intentionally stay narrow:
| Component | Responsibility | Example implementations |
|---|---|---|
Reader |
Fetch byte ranges from an upstream object | HTTP, S3, Hugging Face, local files |
Writer |
Store and retrieve content-addressed chunks | Local disk, object storage, distributed caches |
Metadata |
Track object coverage and chunk lifecycle | Redis, another key-value store, embedded state |
ReaderRegistry |
Route canonical object paths to readers | Application-defined source schemes |
Bring the systems that fit your workload; SparseIO coordinates the read path, sparse coverage, in-flight work, and CAS materialization.
- Sparse by default: requesting one range never requires materializing the whole object.
- Backend agnostic: sources, chunk storage, and metadata are replaceable.
- Runtime neutral: the library remains usable from Tokio, smol, async-std, and other executors.
- Safe under concurrency: overlapping requests share work instead of multiplying upstream traffic.
- Cache, not custody: missing or expired cached chunks fall back to the source of truth.
SparseIO is under active development. The core traits and architecture are taking shape, but the read path, backend integrations, service infrastructure, and public API are not yet ready for production use. The Redis/RESP interface and Owl-inspired in-memory peer and tracker implementations are planned work. Feedback from storage, data infrastructure, and ML systems builders is welcome while these interfaces are still evolving.