Repository navigation
Conversation
The extra_dep target depends on the _extras target of a package with the requested extra set in a private build setting and the extra_select target selects the target of that extra and resets the setting, so that the targets of the extras are analyzed in the same configuration. The extras that the package does not provide fall back to the package itself, like pip.
The dependencies on the extras of other packages from the wheel METADATA, e.g. 'Requires-Dist: bar[socks]', used the '[socks]' target of the hub, which exists only if the extra is requested in the lock file, so the build failed if the lock file did not have the extra. Each whl repository now has an '_extras' target that selects the target of the extra requested via an extra_dep target, and the hub forwards it to the whl repository of the target platform. The extras that the package does not provide fall back to the package, like pip does.
…ries The whl repositories resolve the extras of their dependencies without the hub, so the config.bzl of the hub lists only the packages again and the missing_extra_error rule is removed. This means that the BUILD files of the whl repositories do not change when the extras in the lock file change.
…locked Also check that the extras that a package does not provide are ignored in the whl_library integration test.
Collaborator
Author
|
This draft PR was drafted by a Claude Code agent as a design proposal for wiring the extras of the dependencies without relying on the hub repositories or on the lock file containing the extras. The human author (@aignas) handles the review comments on this PR. This is an agent-generated comment, automatically generated. |
…per-extra # Conflicts: # python/private/pypi/BUILD.bazel # python/private/pypi/hub_repository.bzl # python/private/pypi/render_pkg_aliases.bzl
The self edges were matched by comparing the name of the requirement with dashes to the normalized name of the package with underscores, so the extras that include other extras, e.g. `foo-bar[all]` requiring `foo-bar[a,b]`, were never resolved for the packages with more than one word in the name.
The whl repositories did not get the requested extras, so the `pkg` targets did not include them when there is no hub to select the extras, e.g. when using a vendored `requirements.bzl` or `repo_prefix`, and requesting an extra that the package does not provide failed with a missing target. The `requirement` attribute of the whl repository rules is used again. The `pkg` target includes the extras of the requirement as before, the `[]` target is the package without extras and the `[<extra>]` targets are also created for the requested extras that the package does not provide. Such targets are the same as the package without extras, because an unknown extra has no dependencies, which is how `pip` handles them. The `requirements.bzl` file generated by `pip_parse` keeps the extras in the requirements and the hub repositories use the `[]` targets of the whl repositories instead of the `pkg` targets.
The unified `@pypi` hub only had the targets of the packages, so the `[]` and `[<extra>]` targets were only usable through the concrete hubs. Add the `[]` target for all of the packages and the `[<extra>]` targets for the hubs that request the extra, the other hubs get the usual missing package error. The `requirement` macro of the unified hub accepts an extra and the `requirements` macro is added, just like in the concrete hubs.
A lock file may request an extra that the package does not provide, e.g.
`foo[nope]==1.0`. `pip` only warns about it and installs the package
without the extra, but the hub pointed to the `[nope]` target of the whl
repository, which did not exist.
The whl repositories of a hub do not get the requested extras, so that
changing the extras does not refetch them. Instead they read the extras
that are requested in the hub from the `config.bzl` of the hub when the
BUILD file is loaded and create the targets for the extras that the
package does not provide. Such targets are the same as the package
without extras and a warning is printed like `pip` does:
WARNING: foo 1.0 does not provide the extra 'nope'
The same warning is printed if the extra comes from the `requirement`
attribute. The `config.bzl` is now also loaded by the packages without
any dependencies.
The news entry only listed the additions, but the extras of the dependencies, the `whl` targets and the members of the requirement cycles also behave differently, so add a separate entry for the changes. Document how the extras that a package does not provide and the vendored `requirements.bzl` files are handled.
…without-hub # Conflicts: # python/private/pypi/BUILD.bazel # python/private/pypi/render_pkg_aliases.bzl
…to feat.aignas.extras-without-hub The conflicts are resolved by taking the side of the extras targets branch, which means that the wiring of the `extra_dep` targets in the whl repositories and the hub is dropped here and only the rules and their tests are kept. The wiring is redone on top of this merge.
The `extra_dep` and `extra_select` rules used the `pkg` and `actual` attributes for their dependency, so the aspects that propagate along `deps` did not reach the package, and they only forwarded a subset of the providers, e.g. `OutputGroupInfo` was dropped. Use the `deps` attribute and forward all of the providers that the targets of the whl and hub repositories have. The `extra_select` rule also prints a warning when an extra without a target is requested, which is how `pip` handles the unknown extras.
A dependency on an extra of another package, e.g. `pytest-cov` requiring `coverage[toml]`, used the `[toml]` target of the hub, which only exists if the lock file requests the extra, and failed the build otherwise. The lock files often do not have the extras, e.g. `pip-compile` strips them by default, even though the packages of the extras are locked. The hub cannot create the targets for all of the extras, because the extras that a package provides are only known from its `METADATA`. So the whl repository of the dependent package creates an `extra_dep` target for each such dependency and the whl repository of the dependency selects its `[<extra>]` target in the `_extras` target, which the hub forwards to the whl repository of the target platform like the other targets. The dependencies of the extra that are not in the hub are left out like all of the other dependencies that are not in the hub, and an extra that the dependency does not provide is the same as the dependency without extras, a warning is printed for it like `pip` does.
The extras of the dependencies no longer need to be requested in the hub, so nothing creates the targets that fail the build for them.
Collaborator
Author
This was referenced Nov 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With #4490 a dependency on an extra of another package from the wheel
METADATA, e.g.pytest-covrequiringcoverage[toml], uses the[toml]target of the hub, which only exists if the lock file requeststhe extra, and the build fails otherwise. Lock files often do not have
the extras, e.g.
pip-compilestrips them by default, even though thepackages of the extras are locked. Other rule sets and
pipdo not needthe extra to be requested by the user either.
The hub cannot create the targets for every extra, because the extras
that a package provides are only known from its
METADATA, i.e. afterfetching the wheel. So the whl repositories resolve them:
pytest-covcreates anextra_deptarget thatdepends on
@pypi//coverage:_extraswith a privaterequested_extrabuild setting set to
tomlvia a transition.coveragecreates anextra_selecttargetnamed
_extrasthat selects its[toml]target based on the settingand resets the setting on that edge, so the targets of the extras and
their dependencies are analyzed in the same configuration as the rest
of the build.
_extrasto the whl repository of the targetplatform like the other per-package aliases.
Both rules use the
depsattribute, so the aspects that propagate alongdepsreach the package, and they forward all of the providers of theselected target.
The dependencies of the extra that are not in the hub are left out like
all of the other dependencies that are not in the hub. An extra that the
package does not provide is the same as the package without extras and
a warning is printed, which is what
pipdoes. Themissing_extra_errorrule is no longer needed and is removed.
The
[extra]targets of the hub for the extras requested in the lockfile are unchanged.
DO NOT MERGE: awaiting #4490
Work towards #2786