Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Run custom command to download external dependency #19267

Open
gonzojive opened this issue Aug 17, 2023 · 3 comments
Open

Feature Request: Run custom command to download external dependency #19267

gonzojive opened this issue Aug 17, 2023 · 3 comments
Assignees
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) stale Issues or PRs that are stale (no activity for 30 days) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request

Comments

@gonzojive
Copy link
Contributor

Description of the feature request:

I want to use a command like gsutil cp gs://my-bucket/thing.tar /tmp/thing.tar to download the @thing dependency in my workspace. I'd like this to work with the repository cache.

Which category does this issue belong to?

External Dependency

What underlying problem are you trying to solve with this feature?

Two use cases:

  1. Using gsutil to download GCS files as external dependencies in a bazel workspace in a way that is just as performant as http_archive.
  2. Using Coursier with a a private Google Artifact Registry for maven and authenticating with Workload Identity Federation. This can seemingly be done with a plugin to Coursier that handles downloading from artifactregistry:// URLs.

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

https://bazel.build/rules/lib/builtins/repository_ctx#download documents the only mechanism for adding entries to the repository cache using repository_ctx: By downloding them from URLs.

Related issues:

Any other information, logs, or outputs that you want to share?

No response

@gonzojive gonzojive changed the title Feature Request: Run custom command to download external dependencies Feature Request: Run custom command to download external dependency Aug 17, 2023
@Pavank1992 Pavank1992 added the team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. label Aug 17, 2023
@meteorcloudy meteorcloudy added P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed untriaged labels Aug 22, 2023
@Wyverald
Copy link
Member

We're currently working on the "true" repository cache: #12227 This is not exactly what you're asking for, but I imagine it would solve the issue nonetheless?

Strum355 added a commit to sourcegraph/sourcegraph-public-snapshot that referenced this issue Feb 14, 2024
…59879)

Does what it says on the tin

Caveat:
As this doesn't use the built-in downloaders, this probably cant make use of the repository cache. While it won't refetch it every single time (there is _some_ degree of caching), I'm not sure what will cause it to not use the cached one and refresh it. Its a very fast operation though.
See bazelbuild/bazel#19267

## Test plan

`bazel build //internal/database/migration/shared:generate_stitched_migration_graph`
Copy link

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

@github-actions github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Oct 26, 2024
@malt3
Copy link
Contributor

malt3 commented Oct 26, 2024

For your specific problem there is now rules_gcs. It does use the repository cache.
The true repository cache for a more general solution is still being worked on in #12227.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) stale Issues or PRs that are stale (no activity for 30 days) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request
Projects
None yet
Development

No branches or pull requests

7 participants