Skip to content

feat(pypi): add targets for the extras of the packages - #4490

Draft
aignas wants to merge 10 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.target-per-extra
Draft

aignas wants to merge 10 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.target-per-extra

Conversation

@aignas

@aignas aignas commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

The requested extras of a distribution were passed to its whl repository
via the requirement line, so changing the extras refetched the
repository, the same wheel requested with different extras on different
platforms needed separate repositories, and the extras of the
dependencies written in the wheel METADATA (e.g.
Requires-Dist: coverage[toml]) were ignored, silently dropping their
dependencies.

With this change the whl repositories create an [<extra>] target for
every extra in the Provides-Extra METADATA and, with bzlmod, the
requested extras are no longer passed to them. The hub exposes the
extras requested in the lock file, which follows the rules_pycross
convention:

@pypi//google_api_core               # The package with the requested extras
@pypi//google_api_core:[]            # The package without any extras
@pypi//google_api_core:[async_rest]  # The package with the async_rest extra
@pypi//google_api_core:[grpc]        # The package with the grpc extra
  • The requirement macro accepts a single extra, e.g.
    requirement("google-api-core[grpc]"), and the new requirements
    macro returns a list of labels for any number of packages and extras,
    e.g. requirements("google-api-core[async_rest,grpc]", "numpy").
  • The same distribution requested with different extras on different
    platforms uses a single whl repository and the hub selects the extras
    per platform.
  • The dependencies use the extras from the wheel METADATA. If an extra
    is missing in the lock file, building the target fails with a message
    asking to add it to the requirements or the lock file.
  • A requested extra that the wheel does not provide is the same as the
    package without extras and a warning is printed, like pip does it.
  • The whl targets do not use the extras.
  • The members of experimental_requirement_cycles can be extras, e.g.
    foo[bar], so that only the targets of that extra are part of the
    group.
  • The unified @pypi hub has the same targets for the extras.
  • The whl repositories that are used without a hub, i.e. via the
    vendored requirements.bzl or repo_prefix of the WORKSPACE
    pip_parse, still get the extras via the requirement line and their
    pkg target includes them as before. The dependencies between such
    repositories use the pkg targets and ignore the extras from the
    METADATA, as before.
  • The self-referencing extras, e.g. foo-bar[all] requiring
    foo-bar[a,b], are now resolved for the packages with -, _ or
    . in their names.

Fixes #2786
Fixes #4168

@aignas
aignas requested a review from rickeylev as a code owner November 13, 2026 08:59
@aignas

aignas commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was written by an AI coding agent (Claude Code) working under the direction of @aignas, who reviewed the design and the changes. Review comments will be handled by @aignas, possibly with the help of the agent; any agent-drafted replies or follow-up commits are reviewed by @aignas before being posted. An agent monitors CI only to retry flaky network failures.

This comment was generated by an AI agent and reviewed by the user.

@aignas

aignas commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

FYI, AI is warning

BREAKING CHANGE: The dependencies on the extras of other packages from the
wheel METADATA are now honoured and the build fails with a helpful message
if such an extra is missing in the lock file, e.g. when the requirements are
compiled without --no-strip-extras. The whl targets no longer include the
dependencies of the extras.

However, I am sure that without --no-strip-extras we would fail, because the whl_library would not know what to do. So I am disregarding this as being too cautious.

@aignas aignas changed the title feat(pypi)!: add targets for the extras of the packages feat(pypi): add targets for the extras of the packages Nov 13, 2026
@aignas
aignas marked this pull request as draft November 20, 2026 01:42
@aignas
aignas changed the base branch from main to feat.aignas.extras-helpers November 22, 2026 05:28
@aignas
aignas force-pushed the feat.aignas.target-per-extra branch from 3c48036 to b29c3c6 Compare November 22, 2026 05:40
@aignas
aignas changed the base branch from feat.aignas.extras-helpers to main November 22, 2026 05:40
…per-extra

# Conflicts:
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
@aignas
aignas marked this pull request as ready for review November 22, 2026 07:03
@aignas
aignas marked this pull request as draft November 22, 2026 07:03
The self edges were matched by comparing the name of the requirement
with dashes to the normalized name of the package with underscores, so
the extras that include other extras, e.g. `foo-bar[all]` requiring
`foo-bar[a,b]`, were never resolved for the packages with more than one
word in the name.
The whl repositories did not get the requested extras, so the `pkg`
targets did not include them when there is no hub to select the extras,
e.g. when using a vendored `requirements.bzl` or `repo_prefix`, and
requesting an extra that the package does not provide failed with a
missing target.

The `requirement` attribute of the whl repository rules is used again.
The `pkg` target includes the extras of the requirement as before, the
`[]` target is the package without extras and the `[<extra>]` targets
are also created for the requested extras that the package does not
provide. Such targets are the same as the package without extras,
because an unknown extra has no dependencies, which is how `pip` handles
them. The `requirements.bzl` file generated by `pip_parse` keeps the
extras in the requirements and the hub repositories use the `[]` targets
of the whl repositories instead of the `pkg` targets.
The unified `@pypi` hub only had the targets of the packages, so the
`[]` and `[<extra>]` targets were only usable through the concrete hubs.

Add the `[]` target for all of the packages and the `[<extra>]` targets
for the hubs that request the extra, the other hubs get the usual
missing package error. The `requirement` macro of the unified hub
accepts an extra and the `requirements` macro is added, just like in the
concrete hubs.
A lock file may request an extra that the package does not provide, e.g.
`foo[nope]==1.0`. `pip` only warns about it and installs the package
without the extra, but the hub pointed to the `[nope]` target of the whl
repository, which did not exist.

The whl repositories of a hub do not get the requested extras, so that
changing the extras does not refetch them. Instead they read the extras
that are requested in the hub from the `config.bzl` of the hub when the
BUILD file is loaded and create the targets for the extras that the
package does not provide. Such targets are the same as the package
without extras and a warning is printed like `pip` does:

    WARNING: foo 1.0 does not provide the extra 'nope'

The same warning is printed if the extra comes from the `requirement`
attribute. The `config.bzl` is now also loaded by the packages without
any dependencies.
The news entry only listed the additions, but the extras of the
dependencies, the `whl` targets and the members of the requirement
cycles also behave differently, so add a separate entry for the changes.
Document how the extras that a package does not provide and the vendored
`requirements.bzl` files are handled.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support extras and dependency_groups in the PEP508 env marker evaluation pip.parse drops extras declared on the wheel's dependency (package[extra])

1 participant