Skip to content

Commit

Permalink
Add support for chunking of blobs, using SHA256TREE
Browse files Browse the repository at this point in the history
Buildbarn has invested heavily in using virtual file systems. Both on
the worker and client side it's possible to lazily fault in data from
the CAS. As Buildbarn implements checksum verification where needed,
randomly accessing large files may be slow. To address this, this change
adds support for composing and decomposing CAS objects, using newly
added ConcatenateBlobs() and SplitBlobs() operations.

If implemented naively (e.g., using SHA-256), these operations would not
be verifiable. To rephrase: when merely given the checksum of smaller
objects, there is no way to obtain that of its concatenated version.
This is why we suggest that these operations are only used in
combination with SHA256TREE (see #235).

With these new operations present, there is no true need to use the
Bytestream protocol any longer. Writes can be performed by uploading
smaller parts through BatchUpdateBlobs(), followed by calling
ConcatenateBlobs(). Conversely, reads of large objects can be performed
by calling SplitBlobs() and downloading individual parts through
BatchReadBlobs(). For compatibility, we still permit the Bytestream
protocol to be used. This is a decision we can revisit in REv3.
  • Loading branch information
EdSchouten committed Dec 26, 2022
1 parent 6473a0f commit 4c54573
Show file tree
Hide file tree
Showing 2 changed files with 1,501 additions and 609 deletions.
Loading

0 comments on commit 4c54573

Please sign in to comment.