Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve buildbarn documentation #179

Closed
wants to merge 2 commits into from
Closed

Improve buildbarn documentation #179

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a663a93
chore: fix typos
stagnation Apr 13, 2023
5e23235
Improve documentation for the Buildbarn example
stagnation Apr 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/remote_execution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Keys supported include:
* `engine_address` - address to your RE's engine.
* `action_cache_address` - address to your action cache endpoint.
* `cas_address` - address to your content-addressable storage (CAS) endpoint.
* `tls_ca_certs` - path to a CA certificates bundle. This must be PEM-encoded. If none is set, a default bundle will be used. This path contain environment variables using shell interpolation syntax (i.e. $VAR). They will be substituted before reading the file.
* `tls_ca_certs` - path to a CA certificates bundle. This must be PEM-encoded. If none is set, a default bundle will be used. This path contains environment variables using shell interpolation syntax (i.e. $VAR). They will be substituted before reading the file.
* `tls_client_cert` - path to a client certificate (and intermediate chain), as well as its associated private key. This must be PEM-encoded. This path can contain environment variables using shell interpolation syntax (i.e. $VAR). They will be substituted before reading the file.
* `http_headers` - HTTP headers to inject in all requests to RE. This is a comma-separated list of `Header: Value` pairs. Minimal validation of those headers is done here. This can contain environment variables using shell interpolation syntax ($VAR). They will be substituted before reading the file.

Expand Down
10 changes: 4 additions & 6 deletions examples/remote_execution/buildbarn/.buckconfig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,12 @@
root = .
prelude = prelude

[buck2]
digest_algorithms = SHA256
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tangentially related thought, shouldn't buck2 pick a suitable algorithm that matches one of the supported ones from the REV2 GetCapabilities call?
https://github.com/bazelbuild/remote-apis/blob/35aee1c4a4250d3df846f7ba3e4a4e66cb014ecd/build/bazel/remote/execution/v2/remote_execution.proto#L1960

Copy link
Contributor

@krallin krallin Apr 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, though I think it's a bit unlikely that we'd make this a priority — we've got no use for it internally is a small reason, but the bigger reason would be be that this is a pretty substantial overhaul of how we start up things, since we need this config very early on, which is not true of the RE client, for ultimately fairly little payoff (at least at the moment, all RE providers we're aware of support SHA256, which is the default we use in the OSS build).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment that's true, there's not much gain in consuming GetCapabilities only to look at which digest function to use when SHA256 is pretty much universally supported.

But looking forward a little, the RE api recently adopted SHA256TREE as a digest function which is a precursor to add new functionality to the remote api ConcatenateBlobs and SplitBlobs. Those will allow partial transfer of larger blobs and ultimately become a more efficient replacement of bytestream.write.

There is of course no hurry, I just thought it might be worth poking the project on a trajectory that meshes well with how the protocol is moving.

Copy link
Contributor

@krallin krallin Apr 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But looking forward a little, the RE api recently adopted bazelbuild/remote-apis#235 as a digest function which is a precursor to add new functionality to the remote api bazelbuild/remote-apis#233. Those will allow partial transfer of larger blobs and ultimately become a more efficient replacement of bytestream.write.

Sure, my point there is primarily that actually configuring the hashing scheme you use is rather the trivial bit, there's a lot more to actually implementing describing what you're doing than inferring the configuration.


[buck2_re_client]
action_cache_address = http://localhost:8980
engine_address = http://localhost:8980
cas_address = http://localhost:8980
action_cache_address = grpc://localhost:8980
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather keep this as http://.

Neither of those are actually valid as per the GRPC spec for URIs, but grpc:// appears to be more used for when TLS is desired, so I intend to change #174 to respect that (at which point this would break)

Copy link
Contributor

@krallin krallin Apr 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, let's keep this as-is, and add a separate config field for TLS. I'll update #174 with that.

engine_address = grpc://localhost:8980
cas_address = grpc://localhost:8980
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add tls = false here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, though you already did in the commit that added it.

tls = false
instance_name = fuse
Copy link
Author

@stagnation stagnation May 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record: inline toml comments are parsed as part of the string.
I presume this is a toml file, but you can disregard this if inline comments are not meant to work.

Internal error (stage: remote_call_error): Remote Execution Error (GRPC-SESSION-ID): RE: execution with digest 9a8db5f5c0ba0b07866989dafa06b9914c25483dfedba2b0f53d06251191bdfc:142: Failed to start remote execution: status: Unavailable, message: "No workers exist for instance name prefix \"fuse # or \\\"hardlinking\\\"\" platform {\"properties\
":[{\"name\":\"OSFamily\",\"value\":\"Linux\"},{\"name\":\"container-image\",\"value\":\"docker://ghcr.io/catthehacker/ubuntu:act22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448\"}]}", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

"No workers exist for instance name prefix \"fuse # or \\\"hardlinking\\\"\"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not really a toml file, no. I thin kit'd be good if we used a more robust format for this, but it's not the case currently


[build]
execution_platforms = root//platforms:platforms
65 changes: 49 additions & 16 deletions examples/remote_execution/buildbarn/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,30 +1,63 @@
## Remote execution integration with Build Barn
## Remote execution integration with Buildbarn

This project provides a small example of what a project that utilizes [Build Barn](https://github.com/buildbarn) for RE might look like.
This project provides a small example of what a project that utilizes [Buildbarn](https://github.com/buildbarn).

In this document, we will go over the key configs used in this setup.
Using a local docker-compose deployment from the [example deployment repo](https://github.com/buildbarn/bb-deployments).
If you already have a Buildbarn deployment you can skip that.

### Relevant configs in .buckconfig
### Deploy a local Buildbarn

First, the Build Barn endpoint and certificate should be configured as the following:
```
... $ git clone https://github.com/buildbarn/bb-deployments
... $ cd bb-deployments
.../bb-deployments $ cd docker-compose
.../bb-deployments/docker-compose $ ./run.sh
```

```ini
[buck2_re_client]
engine_address = $BB_ENDPOINT
action_cache_address = $BB_ENDPOINT
cas_address = $BB_ENDPOINT
This uses `docker-compose` to spin up the required Buildbarn services.
Using FUSE based workers, those are generally the fastest as they can load action files on demand
and avoids the overhead of setting up the full input root up front.
In practice many actions do not read all the files in the input root.

If you do not want FUSE workers you can instead switch to hardlinking workers
The example deployments have two worker kinds "fuse", and "hardlinking",
you can see the queues in the Buildbarn scheduler, http://localhost:7982.

```
Buildbarn Scheduler

...

Instance name Platform properties Size Timeo
prefix class

"fuse" OSFamily="Linux" container-image="docker://ghcr.io/catthehacker/ 0 ∞
ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448"
"hardlinking" OSFamily="Linux" container-image="docker://ghcr.io/catthehacker/ 0 ∞
ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448"
```

If you're using the Docker-compose setup, the endpoint would be `http://localhost:8980` for all 3 of those.
More information is available in the [repo](https://github.com/buildbarn/bb-deployments).

### Relevant configs in .buckconfig

First, the Buildbarn endpoint should be configured as the following:

Additionally, set the `digest_algorithm` config to `SHA256`.
```ini
[buck2]
digest_algorithms = SHA256
[buck2_re_client]
engine_address = grpc://localhost:8980
action_cache_address = grpc://localhost:8980
cas_address = grpc://localhost:8980
tls = false

# Select the instance name, "fuse" or "hardlinking", without quotes
instance_name = fuse
```

### Relevant configs in `ExecutionPlatformInfo`
TLS is not used in this example.

Build Barn takes in a Docker image and OSFamily in its RE properties to select a worker.
### Relevant configs in `ExecutionPlatformInfo`

The execution platform used in this project `root//platforms:platforms` do so.
Buildbarn takes in a Docker image and `OSFamily` in its RE properties to select a worker.
This is configured in `root//platforms:platforms`.
2 changes: 1 addition & 1 deletion examples/remote_execution/buildbarn/platforms/defs.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def _platforms(ctx):
local_enabled = True,
remote_enabled = True,
use_limited_hybrid = True,
# Set those up based on what workers you've registered with Build Barn.
# Set those up based on what workers you've registered with Buildbarn.
remote_execution_properties = {
"OSFamily": "Linux",
"container-image": "docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448",
Expand Down
2 changes: 1 addition & 1 deletion examples/remote_execution/engflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Remote execution integration with EngFlow

This project provides a small example of what a project that utilizies [EngFlow](https://www.engflow.com/)'s RE offering might look like.
This project provides a small example of what a project that utilizes [EngFlow](https://www.engflow.com/)'s RE offering might look like.

In this document, we will go over the key configs used in this setup.

Expand Down