Skip to content

feat(pypi): resolve the extras of the dependencies in the whl repositories and expose the pyproject.toml extras - #40

Closed
aignas wants to merge 70 commits into
feat.aignas.target-per-extrafrom
feat.aignas.extras-without-hub
Closed

aignas wants to merge 70 commits into
feat.aignas.target-per-extrafrom
feat.aignas.extras-without-hub

Conversation

@aignas

@aignas aignas commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

With bazel-contrib#4490 a dependency on an extra of another
package from the wheel METADATA, e.g. pytest-cov requiring
coverage[toml], uses the [toml] target of the hub, which only exists
if the lock file requests the extra, and the build fails otherwise. Lock
files often do not have the extras, e.g. pip-compile strips them by
default, even though the packages of the extras are locked.

The hub cannot create the targets for every extra, because the extras
that a package provides are only known from its METADATA, so the whl
repositories resolve them:

  • The whl repository of pytest-cov creates an extra_dep target that
    depends on @pypi//coverage:_extras with a private requested_extra
    build setting set to toml via a transition.
  • The whl repository of coverage creates an extra_select target
    named _extras that selects its [toml] target based on the setting
    and resets the setting on that edge, so the targets of the extras are
    analyzed in the same configuration as the rest of the build.
  • The hub only forwards _extras to the whl repository of the target
    platform. Without a hub, e.g. with a vendored requirements.bzl, the
    extra_dep target depends on the _extras target of the whl
    repository directly.

An extra that the package does not provide is the same as the package
without extras and a warning is printed, which is what pip does, so
bazel query and bazel cquery work across such edges. The
missing_extra_error rule is no longer needed and is removed.

The extras that the user depends on directly can now come from the
pyproject.toml selected with pip.configure, e.g. requests[socks]
in [project] dependencies. They are added to the hub for the target
platforms that match the environment markers of the requirement, even
if the lock file does not request them, so @pypi//requests:[socks]
exists without changing how the lock file is compiled.

The dependencies from the METADATA that are not in the hub are still
left out, which is only correct if they are not needed on any of the
target platforms, e.g. pywin32; sys_platform == "win32" in a hub for
Linux. The hub now exports the marker environments of the target
platforms that have requirements and the whl repositories print a
warning that names the package, the missing dependency and its marker
if the marker can be true on any of them, e.g. when the lock file is
compiled for Python 3.12 and the hub is also used with Python 3.10,
where coverage[toml] needs tomli. The warnings about the
dependencies of an extra are only printed if the extra is used, i.e. it
is requested in the lock file or in the pyproject.toml, or another
package depends on it. They are not printed when using pip_parse in
WORKSPACE.

Research PR, moved from bazel-contrib#4523 and including
bazel-contrib#4522. Based on the branch of
bazel-contrib#4490.

Work towards bazel-contrib#2786

aignas added 30 commits November 8, 2026 02:07
…oups

Read the dependencies, the optional dependencies and the PEP 735
dependency groups from the pyproject.toml and add a helper to select the
packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set,
the aliases of the packages that are not exposed are only visible to the
wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub
exposes to the ones listed in the dependency groups of a `pyproject.toml`
file. The other packages of the hub are still used by the wheels that need
them, but their aliases are not public.
Otherwise the unified hub would point to the hidden aliases of the hub and
fail with a visibility error instead of the missing package error.
pip.configure with ["*", "transitive"] used the packages that are
available on all platforms, which made the platform-specific packages
private. Treat it as if the hub was not configured instead, which keeps
these packages public and out of all_requirements as before.
…igure

Replace the nested comprehensions and the long inline conditions with
explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the
selection is tested in hub_exposure_tests, and test that the pip.default
pyproject_toml is used. Also link the issue in the news entry.
aignas and others added 17 commits November 20, 2026 00:32
…oject.toml

The dependency_groups default is now empty, so a pip.configure tag only
needs a pyproject.toml when it selects dependency groups. A pyproject_toml
set on a tag is used by the later tags of the same module and hub.
… into feat.aignas.pip-configure-extras

# Conflicts:
#	docs/pypi/download.md
#	python/private/pypi/dependency_groups.bzl
#	python/private/pypi/extension.bzl
#	tests/pypi/hub_exposure/hub_exposure_tests.bzl
…symlinks (bazel-contrib#4571)

`test_replaces_dangling_package_link_before_creating_child_link` needs
to create symlinks, so it fails on Windows without the privilege to
create them, which is the case bazel-contrib#4509 is for. Skip it there. The
junction case still runs.

Tested on Windows 11 26H2 with Developer Mode off: the symlink case is
skipped and the other two cases pass. On `main`, the symlink case
fails. The tests also pass on Linux.
…without-hub

# Conflicts:
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/render_pkg_aliases.bzl
…to feat.aignas.extras-without-hub

The conflicts are resolved by taking the side of the extras targets
branch, which means that the wiring of the `extra_dep` targets in the
whl repositories and the hub is dropped here and only the rules and
their tests are kept. The wiring is redone on top of this merge.
The `extra_dep` and `extra_select` rules used the `pkg` and `actual`
attributes for their dependency, so the aspects that propagate along
`deps` did not reach the package, and they only forwarded a subset of
the providers, e.g. `OutputGroupInfo` was dropped.

Use the `deps` attribute and forward all of the providers that the
targets of the whl and hub repositories have. The `extra_select` rule
also prints a warning when an extra without a target is requested, which
is how `pip` handles the unknown extras.
A dependency on an extra of another package, e.g. `pytest-cov` requiring
`coverage[toml]`, used the `[toml]` target of the hub, which only exists
if the lock file requests the extra, and failed the build otherwise.
The lock files often do not have the extras, e.g. `pip-compile` strips
them by default, even though the packages of the extras are locked.

The hub cannot create the targets for all of the extras, because the
extras that a package provides are only known from its `METADATA`. So
the whl repository of the dependent package creates an `extra_dep`
target for each such dependency and the whl repository of the
dependency selects its `[<extra>]` target in the `_extras` target, which
the hub forwards to the whl repository of the target platform like the
other targets.

The dependencies of the extra that are not in the hub are left out like
all of the other dependencies that are not in the hub, and an extra that
the dependency does not provide is the same as the dependency without
extras, a warning is printed for it like `pip` does.
The extras of the dependencies no longer need to be requested in the
hub, so nothing creates the targets that fail the build for them.
…figure-extras

# Conflicts:
#	docs/pypi/download.md
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/dependency_groups.bzl
#	python/private/pypi/extension.bzl
#	python/private/pypi/hub_exposure.bzl
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
#	tests/pypi/dependency_groups/dependency_groups_tests.bzl
#	tests/pypi/extension/extension_tests.bzl
#	tests/pypi/hub_exposure/hub_exposure_tests.bzl
…to feat.aignas.pip-configure-extras

Together with the merge of `upstream/main` this replaces the old copies
of the `pip.configure` and the extras targets changes on this branch
with the current ones. The conflicts are resolved by taking the other
side, so the code that adds the `pyproject.toml` extras to the hubs is
dropped here and is redone on top of this merge.
`select_dependency_groups` now also returns the extras that the selected
requirements request, e.g. `requests[socks]`, so that `pip.configure`
can add them to the hub.

The extras are keyed by the environment markers of the requirements,
including the markers of the self references that select the group of
the requirement, so that the extras can be added only for the target
platforms that request them.
The extras requested in the selected `pyproject.toml` dependency groups
are added to the hub, so `@pypi//requests:[socks]` exists and
`@pypi//requests` includes the extra even if the lock file does not
request it.

`hub_exposure` now returns the extras together with the exposed packages
and `add_extras` adds them to the config settings of the hub `whl_map`.
The environment markers of the requirements are evaluated for each
target platform using the marker environments that the hub builder now
returns, like it is done for the requirements files. A requirement like
`foo[bar]; sys_platform == "win32"` only adds the extra for Windows and
the config setting of the whl repository is split if its platforms
request different extras.

The whl repositories get the requested extras from the hub, so an extra
that the package does not provide has a target that is the same as the
package without extras and a warning is printed.
…he target platforms

The dependencies that are not in the `include` list are dropped by
`pep508_deps.deps` without telling the caller. Accept the marker
environments of the target platforms and return the dropped dependencies
with markers that can be true on any of them, so that the caller can warn
about them.
…om the hub

The whl repositories load `config.bzl` of the hub to know which packages
are in the hub. Also export the marker environments of the target platforms
that have requirements, so that the whl repositories can tell if a
dependency that is not in the hub is needed on any of them. The list is
empty in the WORKSPACE mode, where the requirements are only for the host
platform.
…re needed

The dependencies from the wheel `METADATA` that are not in the hub are
left out silently, which is only correct if they are not needed on any of
the target platforms, e.g. `pywin32; sys_platform == "win32"` in a hub
for Linux. If the lock file is compiled for fewer platforms or Python
versions than the hub is used with, then the missing dependency is only
noticed as an `ImportError` at runtime.

Print a warning if the marker of such a dependency can be true on any of
the target platforms of the hub. The warnings about the dependencies of
the package and of the extras requested in the lock file are printed when
the targets of the whl repository are created. The warnings about the
dependencies of the other extras are printed by the `extra_select` target
when another package depends on the extra, otherwise every package with
extras that are not used would print them.
…tories

`bazel query` needs all of the targets that the extras of the dependencies
can resolve to, so use `genquery` on the consumer that depends on an extra
that is provided and on an extra that is not provided by its dependency.
…' into feat.aignas.extras-without-hub

# Conflicts:
#	python/private/pypi/extension.bzl
#	python/private/pypi/hub_builder.bzl
#	python/private/pypi/whl_library_deps_targets.bzl
#	tests/pypi/pkg_aliases/pkg_aliases_test.bzl
#	tests/pypi/whl_library/testdata/packages.bzl
The whl repositories created by `pip_parse` in WORKSPACE or a vendored
`requirements.bzl` use the `@<prefix>_{name}//:{target}` template for
the dependencies, so the `extra_dep` targets depend on the `_extras`
target of the whl repository directly.
…ithout-hub

Keep the extras of the dependencies when there is no hub, because the
extra_dep targets resolve them in the whl repositories.
@aignas aignas changed the title feat(pypi): resolve the extras of the dependencies in the whl repositories feat(pypi): resolve the extras of the dependencies in the whl repositories and expose the pyproject.toml extras Nov 22, 2026
@aignas

aignas commented Nov 22, 2026

Copy link
Copy Markdown
Owner Author

Split into the stack #42 to #48, the progress is tracked in #41.

This is an agent-generated comment unreviewed by the user.

@aignas aignas closed this Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants