Repository navigation
Conversation
…oups Read the dependencies, the optional dependencies and the PEP 735 dependency groups from the pyproject.toml and add a helper to select the packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set, the aliases of the packages that are not exposed are only visible to the wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub exposes to the ones listed in the dependency groups of a `pyproject.toml` file. The other packages of the hub are still used by the wheels that need them, but their aliases are not public.
Otherwise the unified hub would point to the hidden aliases of the hub and fail with a visibility error instead of the missing package error.
pip.configure with ["*", "transitive"] used the packages that are available on all platforms, which made the platform-specific packages private. Treat it as if the hub was not configured instead, which keeps these packages public and out of all_requirements as before.
…igure Replace the nested comprehensions and the long inline conditions with explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the selection is tested in hub_exposure_tests, and test that the pip.default pyproject_toml is used. Also link the issue in the news entry.
Collaborator
Author
|
This PR was drafted by a Claude Code agent as a design proposal for exposing the This is an agent-generated comment, automatically generated. |
Align the new `pip.configure` documentation, docstrings, and Starlark
files with repository style and documentation conventions.
Add `requires-python` to the `docs/pypi/download.md` example, move the
`pip.configure` section to top-level `##` heading order, clarify the
`pyproject_toml` attribute docstrings and `{ref}` links, document helper
arguments/returns and returned `struct` fields, flatten `hub_exposure`
guards, move module-level test constants before functions, and wrap
Starlark lines and `doc` strings to 80 columns.
Restore the `pip.default.pyproject_toml` and `pip.parse.pyproject_toml` attribute docstrings to their original text per review decision.
When a non-aborting `_fail` callback is injected in tests, `select_dependency_groups` and `_expand_dependency_groups` previously continued executing after `_fail(...)`, causing `hub_exposure` to fail on `None` attribute access or return partial results instead of `None`. Return `None` immediately after each `_fail(...)` call in `dependency_groups.bzl`, guard `selection == None` in `hub_exposure.bzl`, and assert `None` return values in error tests.
Single-level self-reference expansion dropped nested extras from composite extras such as `my-project[all]` and added the project name itself to `packages`, and `dependency_groups` type comments omitted list element types. Expand self-referenced optional groups iteratively with a bounded queue in `dependency_groups.bzl` (with unit tests covering nested, bare, and cyclic self-references), and specify parameterized element types in `pyproject_utils.bzl`, `dependency_groups.bzl`, and `render_pkg_aliases.bzl`.
…oject.toml The dependency_groups default is now empty, so a pip.configure tag only needs a pyproject.toml when it selects dependency groups. A pyproject_toml set on a tag is used by the later tags of the same module and hub.
… into feat.aignas.pip-configure-extras # Conflicts: # docs/pypi/download.md # python/private/pypi/dependency_groups.bzl # python/private/pypi/extension.bzl # tests/pypi/hub_exposure/hub_exposure_tests.bzl
…per-extra # Conflicts: # python/private/pypi/BUILD.bazel # python/private/pypi/hub_repository.bzl # python/private/pypi/render_pkg_aliases.bzl
The self edges were matched by comparing the name of the requirement with dashes to the normalized name of the package with underscores, so the extras that include other extras, e.g. `foo-bar[all]` requiring `foo-bar[a,b]`, were never resolved for the packages with more than one word in the name.
The whl repositories did not get the requested extras, so the `pkg` targets did not include them when there is no hub to select the extras, e.g. when using a vendored `requirements.bzl` or `repo_prefix`, and requesting an extra that the package does not provide failed with a missing target. The `requirement` attribute of the whl repository rules is used again. The `pkg` target includes the extras of the requirement as before, the `[]` target is the package without extras and the `[<extra>]` targets are also created for the requested extras that the package does not provide. Such targets are the same as the package without extras, because an unknown extra has no dependencies, which is how `pip` handles them. The `requirements.bzl` file generated by `pip_parse` keeps the extras in the requirements and the hub repositories use the `[]` targets of the whl repositories instead of the `pkg` targets.
The unified `@pypi` hub only had the targets of the packages, so the `[]` and `[<extra>]` targets were only usable through the concrete hubs. Add the `[]` target for all of the packages and the `[<extra>]` targets for the hubs that request the extra, the other hubs get the usual missing package error. The `requirement` macro of the unified hub accepts an extra and the `requirements` macro is added, just like in the concrete hubs.
A lock file may request an extra that the package does not provide, e.g.
`foo[nope]==1.0`. `pip` only warns about it and installs the package
without the extra, but the hub pointed to the `[nope]` target of the whl
repository, which did not exist.
The whl repositories of a hub do not get the requested extras, so that
changing the extras does not refetch them. Instead they read the extras
that are requested in the hub from the `config.bzl` of the hub when the
BUILD file is loaded and create the targets for the extras that the
package does not provide. Such targets are the same as the package
without extras and a warning is printed like `pip` does:
WARNING: foo 1.0 does not provide the extra 'nope'
The same warning is printed if the extra comes from the `requirement`
attribute. The `config.bzl` is now also loaded by the packages without
any dependencies.
The news entry only listed the additions, but the extras of the dependencies, the `whl` targets and the members of the requirement cycles also behave differently, so add a separate entry for the changes. Document how the extras that a package does not provide and the vendored `requirements.bzl` files are handled.
…figure-extras # Conflicts: # docs/pypi/download.md # python/private/pypi/BUILD.bazel # python/private/pypi/dependency_groups.bzl # python/private/pypi/extension.bzl # python/private/pypi/hub_exposure.bzl # python/private/pypi/hub_repository.bzl # python/private/pypi/render_pkg_aliases.bzl # tests/pypi/dependency_groups/dependency_groups_tests.bzl # tests/pypi/extension/extension_tests.bzl # tests/pypi/hub_exposure/hub_exposure_tests.bzl
…to feat.aignas.pip-configure-extras Together with the merge of `upstream/main` this replaces the old copies of the `pip.configure` and the extras targets changes on this branch with the current ones. The conflicts are resolved by taking the other side, so the code that adds the `pyproject.toml` extras to the hubs is dropped here and is redone on top of this merge.
`select_dependency_groups` now also returns the extras that the selected requirements request, e.g. `requests[socks]`, so that `pip.configure` can add them to the hub. The extras are keyed by the environment markers of the requirements, including the markers of the self references that select the group of the requirement, so that the extras can be added only for the target platforms that request them.
The extras requested in the selected `pyproject.toml` dependency groups are added to the hub, so `@pypi//requests:[socks]` exists and `@pypi//requests` includes the extra even if the lock file does not request it. `hub_exposure` now returns the extras together with the exposed packages and `add_extras` adds them to the config settings of the hub `whl_map`. The environment markers of the requirements are evaluated for each target platform using the marker environments that the hub builder now returns, like it is done for the requirements files. A requirement like `foo[bar]; sys_platform == "win32"` only adds the extra for Windows and the config setting of the whl repository is split if its platforms request different extras. The whl repositories get the requested extras from the hub, so an extra that the package does not provide has a target that is the same as the package without extras and a warning is printed.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lock files often do not record the extras that a project requests, e.g.
uv pip compileandpip-compile --strip-extrasdrop them, so with theper-extra hub targets the
@pypi//requests:[socks]target does notexist and
@pypi//requestsdoes not include the extra, even though thepyproject.tomlasks forrequests[socks].With this change
pip.configurereads the extras from the requirementsin the selected
pyproject.tomldependency groups ([project] dependencies,[project.optional-dependencies]and[dependency-groups]) and adds them to the hub:select_dependency_groupsnow also returns the extras of the selectedrequirements keyed by their environment markers,
hub_exposurereturnsthem for each configured hub and
add_extrasadds them to thewhl_config_settingvalues of the hubwhl_map, which the hub alreadyuses to render the
[<extra>]targets and to tell the whl repositorieswhich extras are requested.
The environment markers are evaluated for each target platform of the
hub like it is done for the requirements files, so a requirement like
foo[bar]; sys_platform == "win32"only adds the extra for Windows andthe other platforms do not get its dependencies. The markers of a self
reference, e.g.
my-project[win]; sys_platform == "win32", apply tothe requirements of the group that it selects.
An extra that the package does not provide is the same as the package
without extras and a warning is printed, because the whl repositories
create the targets for all of the extras that the hub requests. With
dependency_groups = ["*", "transitive"]all of the packages staypublic and the extras of all of the groups are added.
DO NOT MERGE: awaiting #4490
Work towards #2786