Skip to content

feat(pypi): expose the pyproject.toml extras with pip.configure - #4522

Closed
aignas wants to merge 60 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.pip-configure-extras
Closed

aignas wants to merge 60 commits into
bazel-contrib:mainfrom
aignas:feat.aignas.pip-configure-extras

Conversation

@aignas

@aignas aignas commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Lock files often do not record the extras that a project requests, e.g.
uv pip compile and pip-compile --strip-extras drop them, so with the
per-extra hub targets the @pypi//requests:[socks] target does not
exist and @pypi//requests does not include the extra, even though the
pyproject.toml asks for requests[socks].

With this change pip.configure reads the extras from the requirements
in the selected pyproject.toml dependency groups ([project] dependencies, [project.optional-dependencies] and
[dependency-groups]) and adds them to the hub:

pip.parse(
    hub_name = "pypi",
    pyproject_toml = "//:pyproject.toml",
    requirements_lock = "//:requirements_lock.txt",
)
pip.configure(
    hub_name = "pypi",
    dependency_groups = ["default", "group:dev"],
)

select_dependency_groups now also returns the extras of the selected
requirements keyed by their environment markers, hub_exposure returns
them for each configured hub and add_extras adds them to the
whl_config_setting values of the hub whl_map, which the hub already
uses to render the [<extra>] targets and to tell the whl repositories
which extras are requested.

The environment markers are evaluated for each target platform of the
hub like it is done for the requirements files, so a requirement like
foo[bar]; sys_platform == "win32" only adds the extra for Windows and
the other platforms do not get its dependencies. The markers of a self
reference, e.g. my-project[win]; sys_platform == "win32", apply to
the requirements of the group that it selects.

An extra that the package does not provide is the same as the package
without extras and a warning is printed, because the whl repositories
create the targets for all of the extras that the hub requests. With
dependency_groups = ["*", "transitive"] all of the packages stay
public and the extras of all of the groups are added.

DO NOT MERGE: awaiting #4490

Work towards #2786

aignas added 30 commits November 8, 2026 02:07
…oups

Read the dependencies, the optional dependencies and the PEP 735
dependency groups from the pyproject.toml and add a helper to select the
packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set,
the aliases of the packages that are not exposed are only visible to the
wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub
exposes to the ones listed in the dependency groups of a `pyproject.toml`
file. The other packages of the hub are still used by the wheels that need
them, but their aliases are not public.
Otherwise the unified hub would point to the hidden aliases of the hub and
fail with a visibility error instead of the missing package error.
pip.configure with ["*", "transitive"] used the packages that are
available on all platforms, which made the platform-specific packages
private. Treat it as if the hub was not configured instead, which keeps
these packages public and out of all_requirements as before.
…igure

Replace the nested comprehensions and the long inline conditions with
explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the
selection is tested in hub_exposure_tests, and test that the pip.default
pyproject_toml is used. Also link the issue in the news entry.
@aignas

aignas commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

This PR was drafted by a Claude Code agent as a design proposal for exposing the pyproject.toml extras through pip.configure. It is stacked on #4395 and #4490, so the diff against main includes those changes. The human author (aignas) will handle the review comments; no agent is monitoring this PR to reply automatically.

This is an agent-generated comment, automatically generated.

aignas and others added 5 commits November 18, 2026 01:21
Align the new `pip.configure` documentation, docstrings, and Starlark
files with repository style and documentation conventions.

Add `requires-python` to the `docs/pypi/download.md` example, move the
`pip.configure` section to top-level `##` heading order, clarify the
`pyproject_toml` attribute docstrings and `{ref}` links, document helper
arguments/returns and returned `struct` fields, flatten `hub_exposure`
guards, move module-level test constants before functions, and wrap
Starlark lines and `doc` strings to 80 columns.
Restore the `pip.default.pyproject_toml` and `pip.parse.pyproject_toml`
attribute docstrings to their original text per review decision.
When a non-aborting `_fail` callback is injected in tests,
`select_dependency_groups` and `_expand_dependency_groups` previously
continued executing after `_fail(...)`, causing `hub_exposure` to fail
on `None` attribute access or return partial results instead of `None`.

Return `None` immediately after each `_fail(...)` call in
`dependency_groups.bzl`, guard `selection == None` in
`hub_exposure.bzl`, and assert `None` return values in error tests.
Single-level self-reference expansion dropped nested extras from
composite extras such as `my-project[all]` and added the project name
itself to `packages`, and `dependency_groups` type comments omitted list
element types.

Expand self-referenced optional groups iteratively with a bounded queue
in `dependency_groups.bzl` (with unit tests covering nested, bare, and
cyclic self-references), and specify parameterized element types in
`pyproject_utils.bzl`, `dependency_groups.bzl`, and
`render_pkg_aliases.bzl`.
…oject.toml

The dependency_groups default is now empty, so a pip.configure tag only
needs a pyproject.toml when it selects dependency groups. A pyproject_toml
set on a tag is used by the later tags of the same module and hub.
… into feat.aignas.pip-configure-extras

# Conflicts:
#	docs/pypi/download.md
#	python/private/pypi/dependency_groups.bzl
#	python/private/pypi/extension.bzl
#	tests/pypi/hub_exposure/hub_exposure_tests.bzl
…per-extra

# Conflicts:
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
The self edges were matched by comparing the name of the requirement
with dashes to the normalized name of the package with underscores, so
the extras that include other extras, e.g. `foo-bar[all]` requiring
`foo-bar[a,b]`, were never resolved for the packages with more than one
word in the name.
The whl repositories did not get the requested extras, so the `pkg`
targets did not include them when there is no hub to select the extras,
e.g. when using a vendored `requirements.bzl` or `repo_prefix`, and
requesting an extra that the package does not provide failed with a
missing target.

The `requirement` attribute of the whl repository rules is used again.
The `pkg` target includes the extras of the requirement as before, the
`[]` target is the package without extras and the `[<extra>]` targets
are also created for the requested extras that the package does not
provide. Such targets are the same as the package without extras,
because an unknown extra has no dependencies, which is how `pip` handles
them. The `requirements.bzl` file generated by `pip_parse` keeps the
extras in the requirements and the hub repositories use the `[]` targets
of the whl repositories instead of the `pkg` targets.
The unified `@pypi` hub only had the targets of the packages, so the
`[]` and `[<extra>]` targets were only usable through the concrete hubs.

Add the `[]` target for all of the packages and the `[<extra>]` targets
for the hubs that request the extra, the other hubs get the usual
missing package error. The `requirement` macro of the unified hub
accepts an extra and the `requirements` macro is added, just like in the
concrete hubs.
A lock file may request an extra that the package does not provide, e.g.
`foo[nope]==1.0`. `pip` only warns about it and installs the package
without the extra, but the hub pointed to the `[nope]` target of the whl
repository, which did not exist.

The whl repositories of a hub do not get the requested extras, so that
changing the extras does not refetch them. Instead they read the extras
that are requested in the hub from the `config.bzl` of the hub when the
BUILD file is loaded and create the targets for the extras that the
package does not provide. Such targets are the same as the package
without extras and a warning is printed like `pip` does:

    WARNING: foo 1.0 does not provide the extra 'nope'

The same warning is printed if the extra comes from the `requirement`
attribute. The `config.bzl` is now also loaded by the packages without
any dependencies.
The news entry only listed the additions, but the extras of the
dependencies, the `whl` targets and the members of the requirement
cycles also behave differently, so add a separate entry for the changes.
Document how the extras that a package does not provide and the vendored
`requirements.bzl` files are handled.
…figure-extras

# Conflicts:
#	docs/pypi/download.md
#	python/private/pypi/BUILD.bazel
#	python/private/pypi/dependency_groups.bzl
#	python/private/pypi/extension.bzl
#	python/private/pypi/hub_exposure.bzl
#	python/private/pypi/hub_repository.bzl
#	python/private/pypi/render_pkg_aliases.bzl
#	tests/pypi/dependency_groups/dependency_groups_tests.bzl
#	tests/pypi/extension/extension_tests.bzl
#	tests/pypi/hub_exposure/hub_exposure_tests.bzl
…to feat.aignas.pip-configure-extras

Together with the merge of `upstream/main` this replaces the old copies
of the `pip.configure` and the extras targets changes on this branch
with the current ones. The conflicts are resolved by taking the other
side, so the code that adds the `pyproject.toml` extras to the hubs is
dropped here and is redone on top of this merge.
`select_dependency_groups` now also returns the extras that the selected
requirements request, e.g. `requests[socks]`, so that `pip.configure`
can add them to the hub.

The extras are keyed by the environment markers of the requirements,
including the markers of the self references that select the group of
the requirement, so that the extras can be added only for the target
platforms that request them.
The extras requested in the selected `pyproject.toml` dependency groups
are added to the hub, so `@pypi//requests:[socks]` exists and
`@pypi//requests` includes the extra even if the lock file does not
request it.

`hub_exposure` now returns the extras together with the exposed packages
and `add_extras` adds them to the config settings of the hub `whl_map`.
The environment markers of the requirements are evaluated for each
target platform using the marker environments that the hub builder now
returns, like it is done for the requirements files. A requirement like
`foo[bar]; sys_platform == "win32"` only adds the extra for Windows and
the config setting of the whl repository is split if its platforms
request different extras.

The whl repositories get the requested extras from the hub, so an extra
that the package does not provide has a target that is the same as the
package without extras and a warning is printed.
@aignas

aignas commented Nov 22, 2026

Copy link
Copy Markdown
Collaborator Author

Closing to reduce the noise here: this is research on top of #4490 and it continues together with #4523 in the fork at aignas#40. It will come back as a new PR once #4490 has landed and the approach is settled.

This is an agent-generated comment unreviewed by the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants