-
Notifications
You must be signed in to change notification settings - Fork 220
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve buildbarn documentation #179
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,14 +2,12 @@ | |
root = . | ||
prelude = prelude | ||
|
||
[buck2] | ||
digest_algorithms = SHA256 | ||
|
||
[buck2_re_client] | ||
action_cache_address = http://localhost:8980 | ||
engine_address = http://localhost:8980 | ||
cas_address = http://localhost:8980 | ||
action_cache_address = grpc://localhost:8980 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd rather keep this as Neither of those are actually valid as per the GRPC spec for URIs, but There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, let's keep this as-is, and add a separate config field for TLS. I'll update #174 with that. |
||
engine_address = grpc://localhost:8980 | ||
cas_address = grpc://localhost:8980 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's add There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, though you already did in the commit that added it. |
||
tls = false | ||
instance_name = fuse | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the record: inline toml comments are parsed as part of the string.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's not really a toml file, no. I thin kit'd be good if we used a more robust format for this, but it's not the case currently |
||
|
||
[build] | ||
execution_platforms = root//platforms:platforms |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,30 +1,63 @@ | ||
## Remote execution integration with Build Barn | ||
## Remote execution integration with Buildbarn | ||
|
||
This project provides a small example of what a project that utilizes [Build Barn](https://github.com/buildbarn) for RE might look like. | ||
This project provides a small example of what a project that utilizes [Buildbarn](https://github.com/buildbarn). | ||
|
||
In this document, we will go over the key configs used in this setup. | ||
Using a local docker-compose deployment from the [example deployment repo](https://github.com/buildbarn/bb-deployments). | ||
If you already have a Buildbarn deployment you can skip that. | ||
|
||
### Relevant configs in .buckconfig | ||
### Deploy a local Buildbarn | ||
|
||
First, the Build Barn endpoint and certificate should be configured as the following: | ||
``` | ||
... $ git clone https://github.com/buildbarn/bb-deployments | ||
... $ cd bb-deployments | ||
.../bb-deployments $ cd docker-compose | ||
.../bb-deployments/docker-compose $ ./run.sh | ||
``` | ||
|
||
```ini | ||
[buck2_re_client] | ||
engine_address = $BB_ENDPOINT | ||
action_cache_address = $BB_ENDPOINT | ||
cas_address = $BB_ENDPOINT | ||
This uses `docker-compose` to spin up the required Buildbarn services. | ||
Using FUSE based workers, those are generally the fastest as they can load action files on demand | ||
and avoids the overhead of setting up the full input root up front. | ||
In practice many actions do not read all the files in the input root. | ||
|
||
If you do not want FUSE workers you can instead switch to hardlinking workers | ||
The example deployments have two worker kinds "fuse", and "hardlinking", | ||
you can see the queues in the Buildbarn scheduler, http://localhost:7982. | ||
|
||
``` | ||
Buildbarn Scheduler | ||
|
||
... | ||
|
||
Instance name Platform properties Size Timeo | ||
prefix class | ||
|
||
"fuse" OSFamily="Linux" container-image="docker://ghcr.io/catthehacker/ 0 ∞ | ||
ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448" | ||
"hardlinking" OSFamily="Linux" container-image="docker://ghcr.io/catthehacker/ 0 ∞ | ||
ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448" | ||
``` | ||
|
||
If you're using the Docker-compose setup, the endpoint would be `http://localhost:8980` for all 3 of those. | ||
More information is available in the [repo](https://github.com/buildbarn/bb-deployments). | ||
|
||
### Relevant configs in .buckconfig | ||
|
||
First, the Buildbarn endpoint should be configured as the following: | ||
|
||
Additionally, set the `digest_algorithm` config to `SHA256`. | ||
```ini | ||
[buck2] | ||
digest_algorithms = SHA256 | ||
[buck2_re_client] | ||
engine_address = grpc://localhost:8980 | ||
action_cache_address = grpc://localhost:8980 | ||
cas_address = grpc://localhost:8980 | ||
tls = false | ||
|
||
# Select the instance name, "fuse" or "hardlinking", without quotes | ||
instance_name = fuse | ||
``` | ||
|
||
### Relevant configs in `ExecutionPlatformInfo` | ||
TLS is not used in this example. | ||
|
||
Build Barn takes in a Docker image and OSFamily in its RE properties to select a worker. | ||
### Relevant configs in `ExecutionPlatformInfo` | ||
|
||
The execution platform used in this project `root//platforms:platforms` do so. | ||
Buildbarn takes in a Docker image and `OSFamily` in its RE properties to select a worker. | ||
This is configured in `root//platforms:platforms`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tangentially related thought, shouldn't buck2 pick a suitable algorithm that matches one of the supported ones from the REV2 GetCapabilities call?
https://github.com/bazelbuild/remote-apis/blob/35aee1c4a4250d3df846f7ba3e4a4e66cb014ecd/build/bazel/remote/execution/v2/remote_execution.proto#L1960
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly, though I think it's a bit unlikely that we'd make this a priority — we've got no use for it internally is a small reason, but the bigger reason would be be that this is a pretty substantial overhaul of how we start up things, since we need this config very early on, which is not true of the RE client, for ultimately fairly little payoff (at least at the moment, all RE providers we're aware of support SHA256, which is the default we use in the OSS build).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the moment that's true, there's not much gain in consuming GetCapabilities only to look at which digest function to use when SHA256 is pretty much universally supported.
But looking forward a little, the RE api recently adopted SHA256TREE as a digest function which is a precursor to add new functionality to the remote api ConcatenateBlobs and SplitBlobs. Those will allow partial transfer of larger blobs and ultimately become a more efficient replacement of bytestream.write.
There is of course no hurry, I just thought it might be worth poking the project on a trajectory that meshes well with how the protocol is moving.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, my point there is primarily that actually configuring the hashing scheme you use is rather the trivial bit, there's a lot more to actually implementing describing what you're doing than inferring the configuration.