perf(pypi): extract wheels once and reuse

[aignas]

Summary:

• Extract wheels in a repository A.
• Then parse METADATA in a repository B and C and reuse A everywhere.
• We also alias the targets for each entry point that is created in whl_library.
• The wheel repository names will be shorter in general that should help on systems
  like Windows, where path length is sometimes an issue.

Limitations:

• If users use the whl_mods API we cannot reuse the extracted whls.
• If users patch wheels, those are not reused.
• Reused only within the context of a single hub repo, we could do better with
  reuse across all of the hubs, but then we need to solve the "how to tell users
  that we need to have a single index per package? how to customize it?" problem.
• If we find the same wheel downloaded from different indexes in the same hub repo
  then we stop reuse for that particular wheel file. This should be rare?

Extra thoughts on the design:

• This is a step towards automatic cycle resolution.
• With uv.lock we can point directly to the extracted sources because
  the dep graph can be passed to the hub repo.

Once #3791 is resolved this should bring reasonable speedups, especially if there the same wheel used in multiple places.

Fixes #2948
Work towards #2530

[gemini-code-assist]

Code Review

This pull request implements an optimized mode for whl_library (gated by the RULES_PYTHON_WHL_LIBRARY_OPTIMIZED environment variable) to allow wheel reuse across different Python versions by omitting the Python version from spoke repository names, generating per-extra targets, and creating explicit aliases in the hub repository. The code review identified several critical issues with this implementation: static aliases in render_pkg_aliases.bzl break multi-version/multi-platform setups, and iterating over extras_info causes aliases to overwrite each other. Additionally, a naming mismatch in extension.bzl between the unified hub alias and the hub repository package name will result in broken aliases, and skipping pkg__extra target generation when there are no extra dependencies leads to build failures.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The _render_extra_alias function generates a static alias pointing to a single spoke repository. However, in a multi-version or multi-platform setup, there are multiple spoke repositories. We should replace this with a multiplatform alias generator that uses select to choose the correct spoke repository based on the active configuration.

def _render_extra_alias_multiplatform(*, name, repo_mapping, target_suffix):
    if type(repo_mapping) == type(""):
        actual_expr = repr("@{}//:{}".format(repo_mapping, target_suffix))
        load_statement = ""
    else:
        actual_dict = {}
        for config_setting, repo_name in repo_mapping.items():
            key = _repr_config_setting(config_setting)
            actual_dict[key] = repr("@{}//:{}".format(repo_name, target_suffix))

        if len(actual_dict) == 1 and list(actual_dict.keys())[0] == repr("//conditions:default"):
            actual_expr = list(actual_dict.values())[0]
        else:
            sorted_pairs = sorted(actual_dict.items())
            actual_expr = "select({\n"
            for k, v in sorted_pairs:
                actual_expr += "        {}: {},\n".format(k, v)
            actual_expr += "    })"

        needs_load = any(["whl_config_setting" in k for k in actual_dict.keys()])
        load_statement = ""
        if needs_load:
            load_statement = 'load("@rules_python//python/private/pypi:whl_config_setting.bzl", "whl_config_setting")\n'

    return """\
{load_statement}package(default_visibility = ["//visibility:public"])

alias(
    name = "pkg",
    actual = {actual_expr},
)

alias(
    name = "whl",
    actual = {actual_expr},
)
""".format(
        load_statement = load_statement,
        actual_expr = actual_expr,
    )

[gemini-code-assist]

In a multi-version or multi-platform setup, whl_extras contains multiple spoke repositories for the same wheel. Iterating over extras_info.items() and writing directly to files causes the alias for one spoke to overwrite the others, breaking multi-version/multi-platform support for extras. We should use the multiplatform alias generator to create a single select-based alias that correctly routes to the appropriate spoke repository.

    # Generate extra alias directories for optimized mode extras.
    for name, extras_info in whl_extras.items():
        normalized = normalize_name(name)
        pkg_aliases = aliases.get(normalized)
        if not pkg_aliases:
            continue

        # Find all unique extras across all repos for this wheel
        unique_extras = {}
        for extra_names in extras_info.values():
            for extra in extra_names:
                unique_extras[extra] = True

        for extra in unique_extras:
            extra_pkg = "{}_{}".format(normalized, extra)
            files["{}/BUILD.bazel".format(extra_pkg)] = _render_extra_alias_multiplatform(
                name = extra_pkg,
                repo_mapping = pkg_aliases,
                target_suffix = "{}__{}".format(normalized, extra),
            )

        # Also create pkg__ alias (no extras)
        no_extras_pkg = "{}__".format(normalized)
        files["{}/BUILD.bazel".format(no_extras_pkg)] = _render_extra_alias_multiplatform(
            name = no_extras_pkg,
            repo_mapping = pkg_aliases,
            target_suffix = normalized,
        )

[gemini-code-assist]

There is a naming mismatch between the unified hub alias name and the hub repository package name. The unified hub alias is generated with double underscores __ (e.g., requests__security), whereas render_pkg_aliases.bzl generates the hub repository package with a single underscore _ (e.g., requests_security). This mismatch will cause the unified hub aliases to point to non-existent packages. Update this to use a single underscore to match the hub repository package naming.

Suggested change

	alias_name = "%s__%s" % (norm_pkg, extra)
	alias_name = "%s_%s" % (norm_pkg, extra)

[gemini-code-assist]

Skipping the generation of the pkg__extra target when there are no extra dependencies (or when all extra dependencies are already present in the base dependencies) will cause the hub repository's alias to point to a non-existent target, leading to build failures. We should always generate the pkg__extra target, even if it only depends on ":pkg", to ensure that the hub aliases remain valid.

[rickeylev]

saw this was draft, so didn't look too thoroughly, just for things that looked like the bot was obviously wrong

[aignas]

Again, ran out of free tokens, so will postpone for another day.

[aignas]

• 2 types of whl_library repos - one with extracted files, one wiring the deps.

[aignas]

So it seems that it is much better to just separate the repositories - one with extracted whl sources, one with the parsed METADATA. This way the change is surgical and very easy to reason about. This was spiked by hand and then vibed and debugged. The build should work but there are many things missing.

TODO:

• split the new repo into a separate file.
• Create a separate function for the BUILD.bazel file generation and add unit tests.
• Update existing hub_builder.bzl unit tests.
• Remove the confabulated "fixes" to make the code easier to read.

[rickeylev]

/review

[aignas]

This should now work.

EDIT: just realized that merging main into this branch should not have affected it.

[aignas]

/review

[aignas]

OK, I personally like the fact that everything is done as part of whl_library macro, which allows us to change less code.

[aignas]

I'll add some unit tests for piece of mind.

[aignas]

Hmmmm. This is an interesting one:

    common: {
        "requirement": "alabaster==0.7.13",
        "index_url": "https://pypi.org/simple/alabaster",
        "urls": ["https://files.pythonhosted.org/packages/64/88/c7083fc61120ab661c5d0b82cb77079fc1429d3f913a456c1c82cf4658f7/alabaster-0.7.13-py3-none-any.whl"],
        "integrity": "sha256-HuGayoAburtbo/XyWORCLfqG+C8+nO+whZsoPN1/YqM=",
        "filename": "alabaster-0.7.13-py3-none-any.whl",
    }
    missing: {
        "envsubst": ["PIP_INDEX_URL"],
    }

I think in this particular case, we could just add the envsubst to the ignore_keys part. However, if the index_url and the urls differ before interpolation with envsubst, but not after, then maybe we should handle this in some particular way... Some ideas:

• Add the extra URL for each wheel.
• For each URL, we try the URL synchronously.
• For each URL we also add the index_url to ensure that we explicitly state which is which.
• We change the URLs to a dict, where the index_url is the key and we use the setdefault to populate it. For any index URL the value should then be the same and we avoid the error.

[aignas]

Extra things that we should do:

• parse the wheel name and do not use the full wheel name but only use: distribution, py_tag, abi_tag, platform_tag, and a little bit of sha256? Since we start using sub-resource shas, maybe we should not use the sha256 at all?

[aignas]

Given the recent refactors, I've did it again in a more surgical way.