Repository navigation
Conversation
…oups Read the dependencies, the optional dependencies and the PEP 735 dependency groups from the pyproject.toml and add a helper to select the packages using dependency group specifiers.
Add the `hide_unexposed_packages` attribute to the hub repository. When set, the aliases of the packages that are not exposed are only visible to the wheel repositories that depend on them.
Add the `pip.configure` tag class that limits the packages that a hub exposes to the ones listed in the dependency groups of a `pyproject.toml` file. The other packages of the hub are still used by the wheels that need them, but their aliases are not public.
Otherwise the unified hub would point to the hidden aliases of the hub and fail with a visibility error instead of the missing package error.
pip.configure with ["*", "transitive"] used the packages that are available on all platforms, which made the platform-specific packages private. Treat it as if the hub was not configured instead, which keeps these packages public and out of all_requirements as before.
…igure Replace the nested comprehensions and the long inline conditions with explicit loops and use types.is_string.
Keep a single wiring test for pip.configure in the extension tests, as the selection is tested in hub_exposure_tests, and test that the pip.default pyproject_toml is used. Also link the issue in the news entry.
…oject.toml The dependency_groups default is now empty, so a pip.configure tag only needs a pyproject.toml when it selects dependency groups. A pyproject_toml set on a tag is used by the later tags of the same module and hub.
… into feat.aignas.pip-configure-extras # Conflicts: # docs/pypi/download.md # python/private/pypi/dependency_groups.bzl # python/private/pypi/extension.bzl # tests/pypi/hub_exposure/hub_exposure_tests.bzl
…symlinks (bazel-contrib#4571) `test_replaces_dangling_package_link_before_creating_child_link` needs to create symlinks, so it fails on Windows without the privilege to create them, which is the case bazel-contrib#4509 is for. Skip it there. The junction case still runs. Tested on Windows 11 26H2 with Developer Mode off: the symlink case is skipped and the other two cases pass. On `main`, the symlink case fails. The tests also pass on Linux.
…without-hub # Conflicts: # python/private/pypi/BUILD.bazel # python/private/pypi/render_pkg_aliases.bzl
…to feat.aignas.extras-without-hub The conflicts are resolved by taking the side of the extras targets branch, which means that the wiring of the `extra_dep` targets in the whl repositories and the hub is dropped here and only the rules and their tests are kept. The wiring is redone on top of this merge.
The `extra_dep` and `extra_select` rules used the `pkg` and `actual` attributes for their dependency, so the aspects that propagate along `deps` did not reach the package, and they only forwarded a subset of the providers, e.g. `OutputGroupInfo` was dropped. Use the `deps` attribute and forward all of the providers that the targets of the whl and hub repositories have. The `extra_select` rule also prints a warning when an extra without a target is requested, which is how `pip` handles the unknown extras.
A dependency on an extra of another package, e.g. `pytest-cov` requiring `coverage[toml]`, used the `[toml]` target of the hub, which only exists if the lock file requests the extra, and failed the build otherwise. The lock files often do not have the extras, e.g. `pip-compile` strips them by default, even though the packages of the extras are locked. The hub cannot create the targets for all of the extras, because the extras that a package provides are only known from its `METADATA`. So the whl repository of the dependent package creates an `extra_dep` target for each such dependency and the whl repository of the dependency selects its `[<extra>]` target in the `_extras` target, which the hub forwards to the whl repository of the target platform like the other targets. The dependencies of the extra that are not in the hub are left out like all of the other dependencies that are not in the hub, and an extra that the dependency does not provide is the same as the dependency without extras, a warning is printed for it like `pip` does.
The extras of the dependencies no longer need to be requested in the hub, so nothing creates the targets that fail the build for them.
…figure-extras # Conflicts: # docs/pypi/download.md # python/private/pypi/BUILD.bazel # python/private/pypi/dependency_groups.bzl # python/private/pypi/extension.bzl # python/private/pypi/hub_exposure.bzl # python/private/pypi/hub_repository.bzl # python/private/pypi/render_pkg_aliases.bzl # tests/pypi/dependency_groups/dependency_groups_tests.bzl # tests/pypi/extension/extension_tests.bzl # tests/pypi/hub_exposure/hub_exposure_tests.bzl
…to feat.aignas.pip-configure-extras Together with the merge of `upstream/main` this replaces the old copies of the `pip.configure` and the extras targets changes on this branch with the current ones. The conflicts are resolved by taking the other side, so the code that adds the `pyproject.toml` extras to the hubs is dropped here and is redone on top of this merge.
`select_dependency_groups` now also returns the extras that the selected requirements request, e.g. `requests[socks]`, so that `pip.configure` can add them to the hub. The extras are keyed by the environment markers of the requirements, including the markers of the self references that select the group of the requirement, so that the extras can be added only for the target platforms that request them.
The extras requested in the selected `pyproject.toml` dependency groups are added to the hub, so `@pypi//requests:[socks]` exists and `@pypi//requests` includes the extra even if the lock file does not request it. `hub_exposure` now returns the extras together with the exposed packages and `add_extras` adds them to the config settings of the hub `whl_map`. The environment markers of the requirements are evaluated for each target platform using the marker environments that the hub builder now returns, like it is done for the requirements files. A requirement like `foo[bar]; sys_platform == "win32"` only adds the extra for Windows and the config setting of the whl repository is split if its platforms request different extras. The whl repositories get the requested extras from the hub, so an extra that the package does not provide has a target that is the same as the package without extras and a warning is printed.
…he target platforms The dependencies that are not in the `include` list are dropped by `pep508_deps.deps` without telling the caller. Accept the marker environments of the target platforms and return the dropped dependencies with markers that can be true on any of them, so that the caller can warn about them.
…om the hub The whl repositories load `config.bzl` of the hub to know which packages are in the hub. Also export the marker environments of the target platforms that have requirements, so that the whl repositories can tell if a dependency that is not in the hub is needed on any of them. The list is empty in the WORKSPACE mode, where the requirements are only for the host platform.
…re needed The dependencies from the wheel `METADATA` that are not in the hub are left out silently, which is only correct if they are not needed on any of the target platforms, e.g. `pywin32; sys_platform == "win32"` in a hub for Linux. If the lock file is compiled for fewer platforms or Python versions than the hub is used with, then the missing dependency is only noticed as an `ImportError` at runtime. Print a warning if the marker of such a dependency can be true on any of the target platforms of the hub. The warnings about the dependencies of the package and of the extras requested in the lock file are printed when the targets of the whl repository are created. The warnings about the dependencies of the other extras are printed by the `extra_select` target when another package depends on the extra, otherwise every package with extras that are not used would print them.
…tories `bazel query` needs all of the targets that the extras of the dependencies can resolve to, so use `genquery` on the consumer that depends on an extra that is provided and on an extra that is not provided by its dependency.
…' into feat.aignas.extras-without-hub # Conflicts: # python/private/pypi/extension.bzl # python/private/pypi/hub_builder.bzl # python/private/pypi/whl_library_deps_targets.bzl # tests/pypi/pkg_aliases/pkg_aliases_test.bzl # tests/pypi/whl_library/testdata/packages.bzl
The whl repositories created by `pip_parse` in WORKSPACE or a vendored
`requirements.bzl` use the `@<prefix>_{name}//:{target}` template for
the dependencies, so the `extra_dep` targets depend on the `_extras`
target of the whl repository directly.
…ithout-hub Keep the extras of the dependencies when there is no hub, because the extra_dep targets resolve them in the whl repositories.
7 tasks done
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With bazel-contrib#4490 a dependency on an extra of another
package from the wheel
METADATA, e.g.pytest-covrequiringcoverage[toml], uses the[toml]target of the hub, which only existsif the lock file requests the extra, and the build fails otherwise. Lock
files often do not have the extras, e.g.
pip-compilestrips them bydefault, even though the packages of the extras are locked.
The hub cannot create the targets for every extra, because the extras
that a package provides are only known from its
METADATA, so the whlrepositories resolve them:
pytest-covcreates anextra_deptarget thatdepends on
@pypi//coverage:_extraswith a privaterequested_extrabuild setting set to
tomlvia a transition.coveragecreates anextra_selecttargetnamed
_extrasthat selects its[toml]target based on the settingand resets the setting on that edge, so the targets of the extras are
analyzed in the same configuration as the rest of the build.
_extrasto the whl repository of the targetplatform. Without a hub, e.g. with a vendored
requirements.bzl, theextra_deptarget depends on the_extrastarget of the whlrepository directly.
An extra that the package does not provide is the same as the package
without extras and a warning is printed, which is what
pipdoes, sobazel queryandbazel cquerywork across such edges. Themissing_extra_errorrule is no longer needed and is removed.The extras that the user depends on directly can now come from the
pyproject.tomlselected withpip.configure, e.g.requests[socks]in
[project] dependencies. They are added to the hub for the targetplatforms that match the environment markers of the requirement, even
if the lock file does not request them, so
@pypi//requests:[socks]exists without changing how the lock file is compiled.
The dependencies from the
METADATAthat are not in the hub are stillleft out, which is only correct if they are not needed on any of the
target platforms, e.g.
pywin32; sys_platform == "win32"in a hub forLinux. The hub now exports the marker environments of the target
platforms that have requirements and the whl repositories print a
warning that names the package, the missing dependency and its marker
if the marker can be true on any of them, e.g. when the lock file is
compiled for Python 3.12 and the hub is also used with Python 3.10,
where
coverage[toml]needstomli. The warnings about thedependencies of an extra are only printed if the extra is used, i.e. it
is requested in the lock file or in the
pyproject.toml, or anotherpackage depends on it. They are not printed when using
pip_parseinWORKSPACE.Research PR, moved from bazel-contrib#4523 and including
bazel-contrib#4522. Based on the branch of
bazel-contrib#4490.
Work towards bazel-contrib#2786