Skip to content

feat(pypi): resolve the extras of the dependencies in the whl repositories - #4523

Closed
aignas wants to merge 43 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.extras-without-hub
Closed

aignas wants to merge 43 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.extras-without-hub

Conversation

@aignas

@aignas aignas commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

With #4490 a dependency on an extra of another package from the wheel
METADATA, e.g. pytest-cov requiring coverage[toml], uses the
[toml] target of the hub, which only exists if the lock file requests
the extra, and the build fails otherwise. Lock files often do not have
the extras, e.g. pip-compile strips them by default, even though the
packages of the extras are locked. Other rule sets and pip do not need
the extra to be requested by the user either.

The hub cannot create the targets for every extra, because the extras
that a package provides are only known from its METADATA, i.e. after
fetching the wheel. So the whl repositories resolve them:

  • The whl repository of pytest-cov creates an extra_dep target that
    depends on @pypi//coverage:_extras with a private requested_extra
    build setting set to toml via a transition.
  • The whl repository of coverage creates an extra_select target
    named _extras that selects its [toml] target based on the setting
    and resets the setting on that edge, so the targets of the extras and
    their dependencies are analyzed in the same configuration as the rest
    of the build.
  • The hub only forwards _extras to the whl repository of the target
    platform like the other per-package aliases.

Both rules use the deps attribute, so the aspects that propagate along
deps reach the package, and they forward all of the providers of the
selected target.

The dependencies of the extra that are not in the hub are left out like
all of the other dependencies that are not in the hub. An extra that the
package does not provide is the same as the package without extras and
a warning is printed, which is what pip does. The missing_extra_error
rule is no longer needed and is removed.

The [extra] targets of the hub for the extras requested in the lock
file are unchanged.

DO NOT MERGE: awaiting #4490

Work towards #2786

The extra_dep target depends on the _extras target of a package with the
requested extra set in a private build setting and the extra_select target
selects the target of that extra and resets the setting, so that the
targets of the extras are analyzed in the same configuration. The extras
that the package does not provide fall back to the package itself, like pip.
The dependencies on the extras of other packages from the wheel METADATA,
e.g. 'Requires-Dist: bar[socks]', used the '[socks]' target of the hub,
which exists only if the extra is requested in the lock file, so the
build failed if the lock file did not have the extra.

Each whl repository now has an '_extras' target that selects the target
of the extra requested via an extra_dep target, and the hub forwards it to
the whl repository of the target platform. The extras that the package
does not provide fall back to the package, like pip does.
…ries

The whl repositories resolve the extras of their dependencies without the
hub, so the config.bzl of the hub lists only the packages again and the
missing_extra_error rule is removed. This means that the BUILD files of the
whl repositories do not change when the extras in the lock file change.
…locked

Also check that the extras that a package does not provide are ignored in
the whl_library integration test.
@aignas

aignas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

This draft PR was drafted by a Claude Code agent as a design proposal for wiring the extras of the dependencies without relying on the hub repositories or on the lock file containing the extras. The human author (@aignas) handles the review comments on this PR.

This is an agent-generated comment, automatically generated.

…per-extra

# Conflicts:
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
The self edges were matched by comparing the name of the requirement
with dashes to the normalized name of the package with underscores, so
the extras that include other extras, e.g. `foo-bar[all]` requiring
`foo-bar[a,b]`, were never resolved for the packages with more than one
word in the name.
The whl repositories did not get the requested extras, so the `pkg`
targets did not include them when there is no hub to select the extras,
e.g. when using a vendored `requirements.bzl` or `repo_prefix`, and
requesting an extra that the package does not provide failed with a
missing target.

The `requirement` attribute of the whl repository rules is used again.
The `pkg` target includes the extras of the requirement as before, the
`[]` target is the package without extras and the `[<extra>]` targets
are also created for the requested extras that the package does not
provide. Such targets are the same as the package without extras,
because an unknown extra has no dependencies, which is how `pip` handles
them. The `requirements.bzl` file generated by `pip_parse` keeps the
extras in the requirements and the hub repositories use the `[]` targets
of the whl repositories instead of the `pkg` targets.
The unified `@pypi` hub only had the targets of the packages, so the
`[]` and `[<extra>]` targets were only usable through the concrete hubs.

Add the `[]` target for all of the packages and the `[<extra>]` targets
for the hubs that request the extra, the other hubs get the usual
missing package error. The `requirement` macro of the unified hub
accepts an extra and the `requirements` macro is added, just like in the
concrete hubs.
A lock file may request an extra that the package does not provide, e.g.
`foo[nope]==1.0`. `pip` only warns about it and installs the package
without the extra, but the hub pointed to the `[nope]` target of the whl
repository, which did not exist.

The whl repositories of a hub do not get the requested extras, so that
changing the extras does not refetch them. Instead they read the extras
that are requested in the hub from the `config.bzl` of the hub when the
BUILD file is loaded and create the targets for the extras that the
package does not provide. Such targets are the same as the package
without extras and a warning is printed like `pip` does:

    WARNING: foo 1.0 does not provide the extra 'nope'

The same warning is printed if the extra comes from the `requirement`
attribute. The `config.bzl` is now also loaded by the packages without
any dependencies.
The news entry only listed the additions, but the extras of the
dependencies, the `whl` targets and the members of the requirement
cycles also behave differently, so add a separate entry for the changes.
Document how the extras that a package does not provide and the vendored
`requirements.bzl` files are handled.
…without-hub

# Conflicts:
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/render_pkg_aliases.bzl
…to feat.aignas.extras-without-hub

The conflicts are resolved by taking the side of the extras targets
branch, which means that the wiring of the `extra_dep` targets in the
whl repositories and the hub is dropped here and only the rules and
their tests are kept. The wiring is redone on top of this merge.
The `extra_dep` and `extra_select` rules used the `pkg` and `actual`
attributes for their dependency, so the aspects that propagate along
`deps` did not reach the package, and they only forwarded a subset of
the providers, e.g. `OutputGroupInfo` was dropped.

Use the `deps` attribute and forward all of the providers that the
targets of the whl and hub repositories have. The `extra_select` rule
also prints a warning when an extra without a target is requested, which
is how `pip` handles the unknown extras.
A dependency on an extra of another package, e.g. `pytest-cov` requiring
`coverage[toml]`, used the `[toml]` target of the hub, which only exists
if the lock file requests the extra, and failed the build otherwise.
The lock files often do not have the extras, e.g. `pip-compile` strips
them by default, even though the packages of the extras are locked.

The hub cannot create the targets for all of the extras, because the
extras that a package provides are only known from its `METADATA`. So
the whl repository of the dependent package creates an `extra_dep`
target for each such dependency and the whl repository of the
dependency selects its `[<extra>]` target in the `_extras` target, which
the hub forwards to the whl repository of the target platform like the
other targets.

The dependencies of the extra that are not in the hub are left out like
all of the other dependencies that are not in the hub, and an extra that
the dependency does not provide is the same as the dependency without
extras, a warning is printed for it like `pip` does.
The extras of the dependencies no longer need to be requested in the
hub, so nothing creates the targets that fail the build for them.
@aignas

aignas commented Nov 22, 2026

Copy link
Copy Markdown
Collaborator Author

Closing to reduce the noise here: this is research on top of #4490 and it continues in the fork at aignas#40. It will come back as a new PR once #4490 has landed and the approach is settled.

This is an agent-generated comment unreviewed by the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant