Skip to content

Commit

Permalink
Clean up terminology and make a couple of minor clarifications to the…
Browse files Browse the repository at this point in the history 
… remote output service spec.

Use the shorter "server" instead of "remote output service" throughout.

Use "output tree" to refer to the entire tree, and "path" to refer to a specific path in it. Similarly, prefer "path" over "file" when the term might also apply to a directory or symbolic link.

Make it clear that the freezing of finalized artifacts is only in effect until the start of the next build.

PiperOrigin-RevId: 662000359
Change-Id: I18cf77a74a3433242913a2d73f42151ffb113cf4
  • Loading branch information
@tjgq @copybara-github
tjgq authored and copybara-github committed Aug 12, 2024
1 parent 3fc4e1d commit 37dbdcf
Showing 1 changed file with 64 additions and 65 deletions.
129 changes: 64 additions & 65 deletions src/main/protobuf/bazel_output_service.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ option java_outer_classname = "BazelOutputServiceProto";
option go_package = "bazeloutputservice";

// The Bazel Output Service may be used by users of the Remote Execution API to
// construct a directory on the local system that contains all output files of a
// build.
// assist Bazel in constructing the output tree: a directory on the local system
// containing all output files produced by a build.
//
// Primitive implementations of this API may simply download files from the
// Content Addressable Storage (CAS) and store them at their designated
Expand All @@ -40,25 +40,24 @@ option go_package = "bazeloutputservice";
// https://groups.google.com/g/bazel-dev/c/lKzENsNd1Do
// https://docs.google.com/document/d/1W6Tqq8cndssnDI0yzFSoj95oezRKcIhU57nwLHaN1qk/edit
service BazelOutputService {
// Clean all data associated with a single output path, so that the
// next invocation of StartBuild() yields an empty output path. This
// MAY be implemented in a way that's faster than removing all of the
// files from the file system manually.
// Clean all data associated with an output tree, so that the next invocation
// of StartBuild() yields an empty output tree. This MAY be implemented in a
// way that's faster than removing the contents manually.
rpc Clean(CleanRequest) returns (CleanResponse);

// Signal that a new build is about to start.
//
// Bazel uses this call to obtain a directory where outputs of the build may
// be stored, called the output path. Based on the parameters provided, server
// may provide an empty output path, or one that has contents from a previous
// build of the same workspace.
// be stored, called the output tree. Based on the parameters provided, the
// server may provide an empty output tree, or one containing the result of
// a previous build of the same workspace.
//
// In case the output path contains data from a previous build, server is
// In case the output tree contains data from a previous build, server is
// responsible for calling ContentAddressableStorage.FindMissingBlobs() for
// all of the objects that are stored remotely. This ensures that these
// objects don't disappear from the Content Addressable Storage while the
// build is running. Any files that are absent MUST be removed from the output
// path and reported through InitialOutputPathContents.modified_path_prefixes,
// objects don't disappear from the Content Addressable Storage during the
// build. Objects that are no longer available MUST be removed from the output
// tree and reported through InitialOutputPathContents.modified_path_prefixes,
// unless the field has been omitted because it would have been too large.
rpc StartBuild(StartBuildRequest) returns (StartBuildResponse);

Expand All @@ -72,23 +71,22 @@ service BazelOutputService {
// further by Bazel or a local build action within the current build.
//
// For each of these paths, the server MAY decide to store a dirty bit,
// initially unset. Subsequent modifications to the contents, or deletion of,
// the file stored at that path, cause the dirty bit to be set. If the server
// chooses to implement the dirty bit, any paths with the dirty bit set MUST
// be reported back to Bazel in the next
// InitialOutputPathContents.modified_path_prefixes for the same workspace.
// initially unset. Any subsequent modification, deletion or recreation of
// that path causes the dirty bit to be set. If the server chooses to store
// the dirty bit, any paths with the dirty bit set MUST be reported back in
// the next InitialOutputPathContents.modified_path_prefixes.
//
// As an alternative to tracking modifications via a dirty bit, a server MAY
// choose to freeze finalized paths, preventing further modifications to the
// files stored there.
// choose to freeze finalized paths, preventing them from being modified
// until the next StartBuildRequest or CleanRequest.
rpc FinalizeArtifacts(FinalizeArtifactsRequest)
returns (FinalizeArtifactsResponse);

// Signal that a build has been completed.
rpc FinalizeBuild(FinalizeBuildRequest) returns (FinalizeBuildResponse);

// Obtain the status of one or more files, directories or symbolic
// links that are stored in the output path.
// Obtain the status of one or more files, directories or symbolic links
// stored in the output tree.
rpc BatchStat(BatchStatRequest) returns (BatchStatResponse);
}

Expand Down Expand Up @@ -119,10 +117,10 @@ message StartBuildRequest {
// Current version: 1 (experimental).
int32 version = 1;

// A client-chosen value for the output service to uniquely identify the
// workspace the build is being started. This value must be set to ensure that
// the remote output service is capable of managing builds for distinct
// workspaces concurrently.
// A client-chosen value that uniquely identifies the workspace where a build
// is being started. Each workspace corresponds to a separately managed output
// tree. This value MAY be used by the server to manage multiple output trees
// concurrently.
//
// Bazel sets this value to the MD5 sum of the absolute path of the output
// base.
Expand All @@ -134,80 +132,81 @@ message StartBuildRequest {

// A client-chosen value that uniquely identifies this build. This value must
// be provided to most other methods to ensure that operations are targeted
// against the right output path. If the server receives a subsequent request
// against the right output tree. If the server receives a subsequent request
// with a non-matching build_id, it SHOULD send back an error response.
//
// Bazel sets this value to --invocation_id.
string build_id = 3;

// Additional arguments to pass depending on how Bazel communicate with the
// Additional arguments to pass depending on how Bazel communicates with the
// Content Addressable Storage.
//
// In case of a REv2 CAS, the type is
// [StartBuildArgs][bazel_output_service_rev2.StartBuildArgs].
google.protobuf.Any args = 4;

// The absolute path at which the remote output service exposes its output
// paths, as seen from the perspective of the client.
// The absolute path at which the server exposes its output tree, as seen from
// the perspective of the client.
//
// This value needs to be provided by the client, because file system
// namespace virtualization may cause this directory to appear at a location
// that differs from the one used by the service.
// This value needs to be provided by the client because file system namespace
// virtualization may cause this directory to appear at different locations
// as seen by the client and the server.
//
// The purpose of this field is to ensure that the remote output service is
// capable of expanding symbolic links containing absolute paths.
// In addition, this field is used to ensure the server is capable of
// expanding symbolic links containing absolute paths into the output tree.
//
// If this is not set, or an empty string, the service must determine where to
// expose its output path and return an absolute path in
// StartBuildResponse.output_path_suffix.
// If unset or empty, the server must determine where to expose its output
// tree and return an absolute path in StartBuildResponse.output_path_suffix.
string output_path_prefix = 5;

// A map of paths on the system that will become symbolic links pointing to
// locations inside the output path. Similar to output_path_prefix, this
// option is used to ensure the remote output service is capable of expanding
// locations inside the output tree. Similar to output_path_prefix, this
// option is used to ensure the server is capable of expanding additional
// symbolic links.
//
// Map keys are absolute paths, while map values are paths that are
// relative to the output path.
// relative to the output tree.
map<string, string> output_path_aliases = 6;
}

message StartBuildResponse {
// If set, the contents of the output path are almost entirely identical on
// If set, the contents of the output tree are almost entirely identical on
// the results of a previous build. This information may be used by Bazel to
// prevent unnecessary scanning of the file system.
//
// The server MUST leave this field unset in case the contents of the output
// path are empty, not based on a previous build, if no tracking of this
// information is performed, or if the number of changes made to the output
// path is too large to be expressed.
// path are empty, not based on a previous build, if no modification tracking
// is performed. It MAY leave it unset if the number of changes made to the
// output path is too large to be expressed.
InitialOutputPathContents initial_output_path_contents = 1;

// A path that the client must append to StartBuildRequest.output_path_prefix
// to obtain the full path at which outputs of the build are stored.
// to obtain the full path at which the output tree is available.
//
// Bazel replaces bazel-out/ with a symlink targeting this path.
string output_path_suffix = 2;
}

message InitialOutputPathContents {
// The identifier of a previously finalized build whose results are stored in
// the output path.
// the output tree.
string build_id = 1;

// Output path prefixes that have been deleted or modified since they were
// finalized. Any path exactly matching or starting with one of these prefixes
// may be assumed to have been modified or deleted.
// Path prefixes relative to StartBuildResponse.output_path that have been
// modified, deleted or recreated since they were finalized. Any path exactly
// matching or starting with one of these prefixes MUST be assumed by Bazel to
// have been modified or deleted. Any other path MAY be assumed by Bazel to
// have remained unchanged since it was last finalized.
//
// In the interest of performance, the server SHOULD only include path
// prefixes that contain at least one of the paths that were finalized by the
// previous build.
// prefixes that contain at least one of the paths that were previously
// finalized.
repeated string modified_path_prefixes = 2;
}

message StageArtifactsRequest {
message Artifact {
// path is relative to StartBuildResponse.output_path.
// Path relative to StartBuildResponse.output_path.
string path = 1;
// Describe how to stage the artifact.
//
Expand Down Expand Up @@ -242,11 +241,11 @@ message StageArtifactsResponse {

message FinalizeArtifactsRequest {
message Artifact {
// path is relative to StartBuildResponse.output_path.
// Path relative to StartBuildResponse.output_path.
string path = 1;
// Expected digest for this path. This allows server to detect if the path
// has been changed after Bazel finished creating the path and the
// corresponding FinalizeArtifactsRequest is processed.
// Expected digest for this path. This allows the server to detect changes
// the path has been changed after the client finished creating the path and
// the corresponding FinalizeArtifactsRequest is processed.
//
// The concrete type of the locator depending on the CAS Bazel connects to.
// In case of a REv2 CAS, the type is
Expand All @@ -269,9 +268,9 @@ message FinalizeBuildRequest {
// The identifier of the build that should be finalized.
string build_id = 1;

// Whether the build completed successfully. The remote output service MAY,
// for example, use this option to apply different retention policies that
// take the outcome of the build into account.
// Whether the build completed successfully. The server MAY, for example, use
// this option to apply different retention policies that take the outcome of
// the build into account.
bool build_successful = 2;
}

Expand All @@ -280,19 +279,19 @@ message FinalizeBuildResponse {
}

message BatchStatRequest {
// The identifier of the build. The remote output service uses this to
// determine which output path needs to be inspected.
// The identifier of the build. The server uses this to determine which output
// tree needs to be inspected.
string build_id = 1;

// Paths whose status is to be obtained. The server MUST canonicalize each
// path using lstat semantics, i.e., all components except the last must be
// resolved if they are symlinks. If a symlink pointing to a location outside
// of the output path is encountered at any point during the canonicalization
// of the output tree is encountered at any point during the canonicalization
// process, the server MAY use the information in
// StartBuildRequest.output_path_aliases map to continue the canonicalization.
//
// Refer to Stat.type for how to handle a situation where canonicalization
// fails due to a symlink pointing to a location outside of the output path.
// fails due to a symlink pointing to a location outside of the output tree.
//
// Path is relative to StartBuildResponse.output_path.
repeated string paths = 2;
Expand All @@ -307,7 +306,7 @@ message BatchStatResponse {

message Stat {
message File {
// The digest of the file.
// The file digest.
//
// The server MAY leave this field unset if it is unable to compute the
// digest.
Expand Down

0 comments on commit 37dbdcf

Please sign in to comment.