Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
3ef53b2
refactor(pypi): select packages from the pyproject.toml dependency gr…
aignas Nov 7, 2026
7557370
refactor(pypi): support hiding the unexposed packages of a hub
aignas Nov 7, 2026
23127e5
feat(pypi): add pip.configure to select the exposed packages of a hub
aignas Nov 7, 2026
6bb7f83
fix(pypi): only add the exposed packages of a configured hub to @pypi
aignas Nov 7, 2026
b31a734
fix(pypi): treat exposing all packages as not configuring the hub
aignas Nov 8, 2026
9d97360
refactor(pypi): use explicit loops and simpler conditions in pip.conf…
aignas Nov 8, 2026
35c0660
test(pypi): test the pip.default pyproject_toml with pip.configure
aignas Nov 8, 2026
a2049ee
Merge branch 'main' into aignas.feat.dep-group-selectors
aignas Nov 9, 2026
2873e71
feat(pypi): add strip_extras to remove the extras from requirement lines
aignas Nov 13, 2026
2f92d4e
feat(pypi): add keep_extras and dependency excludes to pep508_deps
aignas Nov 13, 2026
efc64f0
feat(pypi): add extra_label and group_member label helpers
aignas Nov 13, 2026
552d8f1
feat(pypi): create a target for every provided extra in the whl repos…
aignas Nov 13, 2026
ace0065
feat(pypi): record the requested extras in whl_config_setting
aignas Nov 13, 2026
c0bd9cc
feat(pypi): add the per-platform extra targets to the hub packages
aignas Nov 13, 2026
998146c
feat(pypi): use a single whl repository for the different extras of a…
aignas Nov 13, 2026
dfaa556
feat(pypi): support extras in the requirement macro and add the requi…
aignas Nov 13, 2026
6e77448
refactor(pypi): extract the failing action of missing_package_error
aignas Nov 13, 2026
3284430
feat(pypi): fail the build when a required extra of a dependency is m…
aignas Nov 13, 2026
cca8d1b
feat(pypi): support the extra targets in WORKSPACE
aignas Nov 13, 2026
181ddc9
feat(pypi): support extras as members of the requirement cycle groups
aignas Nov 13, 2026
cf3b4b6
refactor(pypi): pass the requirement line only to the repositories us…
aignas Nov 13, 2026
be422b2
test(pypi): check the extras of the dependencies from the METADATA
aignas Nov 13, 2026
646f270
docs(pypi): document the extra targets
aignas Nov 13, 2026
dafa838
chore(examples): update the vendored requirements.bzl
aignas Nov 13, 2026
6431f8b
fix(pypi): parse the whl METADATA with CRLF line endings
aignas Nov 13, 2026
c2e07df
test(pypi): drop the requirement from the whl repositories in the int…
aignas Nov 13, 2026
9f9fc9f
refactor(pypi): keep the original hub_builder group map and repo mapp…
aignas Nov 13, 2026
a24fac6
refactor(pypi): minimize the changes to the group library generation
aignas Nov 13, 2026
8f57a4f
test(pypi): remove the overlapping extras tests
aignas Nov 13, 2026
330eb71
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas Nov 15, 2026
c488c62
Merge remote-tracking branch 'upstream/main' into feat.aignas.target-…
aignas Nov 15, 2026
65a8f92
Merge remote-tracking branch 'upstream/main' into feat.aignas.target-…
aignas Nov 17, 2026
7765a28
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas Nov 17, 2026
b014361
Merge branch 'feat.aignas.target-per-extra' into feat.aignas.pip-conf…
aignas Nov 17, 2026
df890f2
feat(pypi): collect the extras of the selected dependency groups
aignas Nov 17, 2026
c7b28a6
feat(pypi): expose the pyproject.toml extras with pip.configure
aignas Nov 17, 2026
45c1c82
docs(pypi): document the pip.configure extras
aignas Nov 17, 2026
c54d57d
feat(pypi): add rules to depend on an extra of a package by its name
aignas Nov 17, 2026
b514331
docs(pypi): name the news entry after the PR
aignas Nov 17, 2026
a094169
feat(pypi): wire the extras of the dependencies without the hub
aignas Nov 17, 2026
a055395
refactor(pypi): stop passing the available extras to the whl reposito…
aignas Nov 17, 2026
45555f3
docs(pypi): document that the extras of the dependencies need not be …
aignas Nov 17, 2026
1ffebb8
refactor(pypi): polish pip.configure docs, docstrings, and formatting
rickeylev Nov 19, 2026
a40d1d5
refactor(pypi): restore pyproject_toml attribute docstrings
rickeylev Nov 19, 2026
1291919
refactor(pypi): short-circuit _fail paths with None in group selection
rickeylev Nov 19, 2026
28b48d5
refactor(pypi): expand nested self-extras and refine type comments
rickeylev Nov 19, 2026
0f35bd1
feat(pypi): make pip.configure a builder that does not require a pypr…
aignas Nov 19, 2026
7c58d7b
docs(pypi): rewrap the pip.configure docs
aignas Nov 19, 2026
cb255c9
Merge remote-tracking branch 'origin/aignas.feat.dep-group-selectors'…
aignas Nov 19, 2026
ad8c5cb
test(windows): skip the symlink case of the cached venv test without …
AustinSchuh Nov 22, 2026
f9c8777
Merge remote-tracking branch 'upstream/main' into feat.aignas.extras-…
aignas Nov 22, 2026
09397b5
Merge remote-tracking branch 'origin/feat.aignas.target-per-extra' in…
aignas Nov 22, 2026
4ae37bd
fix(pypi): forward all providers and propagate aspects in extra_dep
aignas Nov 22, 2026
4fb97d1
feat(pypi): resolve the extras of the dependencies in the whl repos
aignas Nov 22, 2026
4c4672d
refactor(pypi): remove the unused missing_extra_error rule
aignas Nov 22, 2026
d611f6e
docs(pypi): document that the extras of the dependencies need not be …
aignas Nov 22, 2026
9e48973
Merge remote-tracking branch 'upstream/main' into feat.aignas.pip-con…
aignas Nov 22, 2026
019e482
Merge remote-tracking branch 'origin/feat.aignas.target-per-extra' in…
aignas Nov 22, 2026
f1a428a
feat(pypi): collect the extras of the selected dependency groups
aignas Nov 22, 2026
b2c276d
feat(pypi): add the pyproject.toml extras to the hubs with pip.configure
aignas Nov 22, 2026
2017d05
docs(pypi): document the pip.configure extras
aignas Nov 22, 2026
47c96e0
Merge branch 'feat.aignas.target-per-extra' into feat.aignas.extras-w…
aignas Nov 22, 2026
72920ed
Merge branch 'feat.aignas.target-per-extra' into feat.aignas.pip-conf…
aignas Nov 22, 2026
0fd20ce
feat(pypi): return the dependencies that are left out but needed on t…
aignas Nov 22, 2026
6d99d5b
feat(pypi): export the marker environments of the target platforms fr…
aignas Nov 22, 2026
99435c2
feat(pypi): warn about the dependencies that are not in the hub but a…
aignas Nov 22, 2026
b95ab16
test(pypi): query the dependencies with extras through the whl reposi…
aignas Nov 22, 2026
9684bdc
Merge remote-tracking branch 'origin/feat.aignas.pip-configure-extras…
aignas Nov 22, 2026
d7601db
test(pypi): cover the extras of the dependencies without a hub
aignas Nov 22, 2026
5d288ed
Merge branch 'feat.aignas.target-per-extra' into feat.aignas.extras-w…
aignas Nov 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/pypi/download.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,29 @@ the `pyproject.toml` selects the optional dependencies of the project. The
packages that are not part of the hub are ignored, so a single `pyproject.toml`
can list tools that are missing from the lock file.

The extras requested in the selected groups are added to the hub even if the
lock file does not request them, for example because the lock file was
generated with `--strip-extras`. With the following `pyproject.toml`:

```toml
[project]
dependencies = [
"requests[socks]",
"keyring[completion]; sys_platform == 'win32'",
]
```

the hub has the `@pypi_hub//requests:[socks]` target and
`@pypi_hub//requests` includes the `socks` extra. The environment markers of
the requirements are evaluated for each target platform of the hub, so
`@pypi_hub//keyring` only includes the `completion` extra on Windows and
`@pypi_hub//keyring:[completion]` is the same as `@pypi_hub//keyring:[]` on
the other platforms. With `["*", "transitive"]` the extras of all of the
groups are added. The dependencies of the extras, such as `pysocks`, still
have to be in the lock file, and an extra that the package does not provide is
ignored with a warning. See [extras](./use.md#extras) for the targets of the
extras.

:::{versionadded} VERSION_NEXT_FEATURE
:::

Expand Down
36 changes: 27 additions & 9 deletions docs/pypi/use.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,32 @@ platforms, for example `foo[bar]` on Linux and `foo[baz]` on macOS, then
`@pypi//foo:[bar]` on macOS is the same as `@pypi//foo:[]`.

The dependencies of the packages use the extras written in the wheel
`METADATA`, so if `foo` depends on `bar[socks]`, then `@pypi//foo` depends on
`@pypi//bar:[socks]`. If the extra is not in the lock file, then building
`@pypi//foo` fails with a message asking you to add `bar[socks]` to the
requirements or the lock file. For example, when compiling the requirements
with `uv pip compile`, pass `--no-strip-extras` to keep the extras in the
output.
`METADATA`, so if `foo` depends on `bar[socks]`, then `@pypi//foo` includes
`bar` and the dependencies of its `socks` extra. The extras of the
dependencies are resolved by the repository of `bar` from its own `METADATA`,
so they do not need to be in the lock file, e.g. when the requirements are
compiled with the extras stripped. This does not change `@pypi//bar`, which
only includes the extras requested in the lock file. The dependencies of the
extra that are not in the lock file are left out like any other dependency
that is not in the lock file, see below for the warning that is printed if
they are needed. If `bar` does not provide the `socks` extra,
then only `bar` is used and a warning is printed when `@pypi//foo` is
analyzed, e.g. `WARNING: bar 1.0 does not provide the extra 'socks'`, which is
how `pip` handles the unknown extras.

The dependencies from the wheel `METADATA` that are not in the lock file are
left out, which is correct if they are not needed on any of the target
platforms of the hub, e.g. `pywin32; sys_platform == "win32"` in a hub that is
only for Linux. If the marker of such a dependency can be true on any of the
target platforms that have requirements, then a warning that names the missing
package is printed, because importing it may fail at runtime. For example, if
the lock file is compiled for Python 3.12 only and the hub is also used with
Python 3.10, where `coverage[toml]` needs `tomli`, then the warning asks to add
`tomli` to the lock file. The fix is to lock for all of the target platforms
and Python versions of the hub. The warnings about the dependencies of an
extra are only printed if the extra is used, i.e. it is requested in the lock
file or another package depends on it. The warnings are not printed when using
`pip_parse` in `WORKSPACE`.

If the lock file requests an extra that the package does not provide, e.g.
`foo[nope]`, then `@pypi//foo:[nope]` is the same as `@pypi//foo:[]`, because
Expand All @@ -142,9 +162,7 @@ When the `requirements.bzl` file is vendored, there is no hub repo and the
labels returned by the `requirement` helper point to the repositories of the
wheels. Their `pkg` targets include the extras from the lock file and they
have the `[]` target and the `[extra]` targets for the extras used in the lock
file and for all of the extras that the package provides. The dependencies
between the packages use the `pkg` targets and ignore the extras from the wheel
`METADATA` like before, so the extras need to be in the lock file.
file and for all of the extras that the package provides.

The `whl` targets, e.g. `@pypi//foo:whl`, do not use the extras and include
only the wheels of the package and its dependencies without extras.
Expand Down
30 changes: 16 additions & 14 deletions news/2786.changed.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
(pypi) The dependencies between the PyPI packages now use the extras from the
wheel `METADATA`, e.g. if `foo` requires `bar[socks]`, then `@pypi//foo`
depends on `@pypi//bar:[socks]` instead of `@pypi//bar`. Previously such extras
were ignored. If the lock file of a hub does not request the extra, the build
fails with a message asking to add it, e.g. by compiling the requirements with
`--no-strip-extras`. This does not apply when there is no hub, e.g. with a
vendored `requirements.bzl`, where the dependencies use the `pkg` targets,
which include the extras from the lock file, as before. The `whl` targets no longer include the wheels of the
dependencies of the extras. A member of `experimental_requirement_cycles`
without extras, e.g. `foo`, is now only the package without extras, so the
cycles that go through `foo[bar]` need `foo[bar]` as a member. The same
distribution that is requested with different extras on different platforms
now uses a single whl repository. The `@pypi//foo` targets still include the
extras requested in the lock file. An extra that the package does not provide
is still ignored, but a warning is now printed like `pip` does, e.g.
`WARNING: foo 1.0 does not provide the extra 'nope'`
includes `bar` and the dependencies of its `socks` extra. Previously such
extras were ignored. The extra does not need to be requested in the lock file
of the hub, because it is resolved by the whl repository of `bar`, and the
dependencies of the extra that are not in the lock file are left out. The
`whl` targets no longer include the wheels of the dependencies of the extras.
A member of `experimental_requirement_cycles` without extras, e.g. `foo`, is
now only the package without extras, so the cycles that go through `foo[bar]`
need `foo[bar]` as a member. The same distribution that is requested with
different extras on different platforms now uses a single whl repository. The
`@pypi//foo` targets still include the extras requested in the lock file. An
extra that the package does not provide is still ignored, but a warning is now
printed like `pip` does, e.g.
`WARNING: foo 1.0 does not provide the extra 'nope'`. A warning is also
printed about the dependencies from the wheel `METADATA` that are left out
because they are not in the lock file, if they are needed on any of the
target platforms of the hub
([#2786](https://github.com/bazel-contrib/rules_python/issues/2786)).
7 changes: 7 additions & 0 deletions news/4522.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
(pypi) {obj}`pip.configure` now adds the extras requested in the selected
`pyproject.toml` dependency groups, e.g. `requests[socks]`, to the hub, so
that the `@pypi//requests:[socks]` target exists and `@pypi//requests`
includes the extra even if the lock file does not request it. The environment
markers of the requirements are evaluated for each target platform, so an
extra is only added for the platforms that request it
([#2786](https://github.com/bazel-contrib/rules_python/issues/2786)).
3 changes: 0 additions & 3 deletions python/private/internal_dev_deps.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,6 @@ def _internal_dev_deps_impl(mctx):
name = "whl_library_extras_pkg",
root = "//tests/pypi/whl_library/testdata/pkg:BUILD.bazel",
output = "pkg-1.0-any-none-any.whl",
# Without a hub the dependencies use the `pkg` target, which includes
# the extras of the requirement.
requirement = "pkg[optional]",
# The following is necessary to enable pipstar and make tests faster
config_load = "@rules_python//tests/pypi/whl_library/testdata:packages.bzl",
dep_template = "@whl_library_extras_{name}//:{target}",
Expand Down
24 changes: 23 additions & 1 deletion python/private/pypi/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
load("//python:py_binary.bzl", "py_binary")
load("//python:py_library.bzl", "py_library")
load("//python/private:distribution_filegroup.bzl", "distribution_filegroup")
load(":extra_dep.bzl", "requested_extra")

package(default_visibility = ["//:__subpackages__"])

Expand All @@ -26,6 +27,13 @@ exports_files(
visibility = ["//visibility:public"],
)

# The extra requested by the `extra_dep` targets in the whl repositories.
requested_extra(
name = "requested_extra",
build_setting_default = "",
visibility = ["//visibility:public"],
)

py_library(
name = "rewriters_lib",
srcs = [
Expand Down Expand Up @@ -110,6 +118,17 @@ bzl_library(
deps = ["//python/private:bazel_tools"],
)

bzl_library(
name = "extra_dep",
srcs = ["extra_dep.bzl"],
deps = [
"//python/private:py_cc_link_params_info",
"//python/private:py_info",
"//python/private:reexports",
"@bazel_skylib//rules:common_settings",
],
)

bzl_library(
name = "env_marker_setting",
srcs = ["env_marker_setting.bzl"],
Expand Down Expand Up @@ -367,6 +386,7 @@ bzl_library(
name = "pkg_aliases",
srcs = ["pkg_aliases.bzl"],
deps = [
":extra_dep",
":labels",
":whl_library_deps_targets",
"//python:py_library",
Expand Down Expand Up @@ -624,8 +644,8 @@ bzl_library(
srcs = ["whl_library_deps_targets.bzl"],
deps = [
":env_marker_setting",
":extra_dep",
":labels",
":missing_package",
":pep508_deps",
"//python:py_library",
"//python/private:normalize_name",
Expand Down Expand Up @@ -667,6 +687,8 @@ bzl_library(
srcs = ["hub_exposure.bzl"],
deps = [
":dependency_groups",
":pep508_evaluate",
":whl_config_setting",
"//python/private:pyproject_utils",
],
)
Expand Down
5 changes: 5 additions & 0 deletions python/private/pypi/config.bzl.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,8 @@ with your usecase. This may change in between rules_python versions without any
"""

packages = %%PACKAGES%%

# The marker environments of the target platforms that have requirements. They
# are used to warn about the dependencies that are not in `packages`, but are
# needed on any of the target platforms.
target_platform_envs = %%TARGET_PLATFORM_ENVS%%
95 changes: 80 additions & 15 deletions python/private/pypi/dependency_groups.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ load("@bazel_skylib//lib:types.bzl", "types")
load("//python/private:normalize_name.bzl", "normalize_name")
load(":pep508_requirement.bzl", "requirement")

# The limit for processing the groups selected by the self references, e.g.
# `my-project[test]`. A group is processed once for each different set of
# markers that it is selected with.
_MAX_SELF_REFERENCES = 10000

def select_dependency_groups(pyproject, specs, _fail = fail):
"""Select the packages named by the given dependency group specifiers.

Expand All @@ -30,6 +35,13 @@ def select_dependency_groups(pyproject, specs, _fail = fail):
* `all`: {type}`bool` whether all of the packages including the
transitive dependencies should be selected, i.e. `"*"` and
`"transitive"` were given.
* `extras`: {type}`dict[str, dict[tuple[str], list[str]]]` the
normalized package names mapped to the extras that the selected
requirements request, e.g. `requests[socks]`. The extras are
keyed by the environment markers that all need to be true for
the extras to be requested, e.g. `("sys_platform == 'win32'",)`
or an empty tuple if there are no markers. When `all` is set,
the extras of all of the groups are returned.
Returns `None` on failure.
"""
groups = {"default": pyproject.dependencies}
Expand All @@ -41,18 +53,17 @@ def select_dependency_groups(pyproject, specs, _fail = fail):
for name, reqs in expanded.items():
groups["group:" + name] = reqs

if "transitive" in specs:
select_all = "transitive" in specs
if select_all:
if "*" not in specs:
_fail("The 'transitive' dependency group specifier is only supported together with '*', which selects all of the packages, got: {}".format(
specs,
))
return None
return struct(
# list[str]: sorted normalized package names selected by `specs`.
packages = [],
# bool: True when all hub packages are selected.
all = True,
)

# All of the packages are selected, but the extras are only known
# from the groups.
specs = ["*"]

selected = {}
for spec in specs:
Expand All @@ -70,30 +81,84 @@ def select_dependency_groups(pyproject, specs, _fail = fail):

# A self reference like `my-project[test]` selects the optional
# dependencies, including nested composite extras.
#
# The groups are queued together with the markers of the self references
# that selected them, because the requirements of such groups are only
# requested when the markers are true. A group is processed again if it is
# selected with markers that are not known to be stricter than the ones
# that it was processed with.
project = normalize_name(pyproject.name or "")
packages = {}
queue = list(selected)
for _ in range(len(groups)):
extras = {}
processed = {}
queue = [(key, ()) for key in selected]
for _ in range(_MAX_SELF_REFERENCES):
if not queue:
break
for req in groups[queue.pop(0)]:
key, markers = queue.pop(0)
if _is_processed(processed.setdefault(key, []), markers):
continue
processed[key].append(markers)
for req in groups[key]:
req = requirement(req)
req_markers = _add_marker(markers, req.marker)
if req.name_ != project:
packages[req.name_] = None
if req.extras:
pkg_extras = extras.setdefault(req.name_, {}).setdefault(req_markers, {})
for extra in req.extras:
pkg_extras[extra] = None
continue
for extra in req.extras:
opt_key = "optional:" + extra
if opt_key in groups and opt_key not in selected:
selected[opt_key] = None
queue.append(opt_key)
if opt_key in groups:
queue.append((opt_key, req_markers))

if queue:
_fail("Too many self references with different markers in the dependency groups of '{}'".format(
pyproject.name,
))
return None

return struct(
# list[str]: sorted normalized package names selected by `specs`.
packages = sorted(packages),
packages = [] if select_all else sorted(packages),
# bool: True when all packages including transitive ones are selected.
all = False,
all = select_all,
# dict[str, dict[tuple[str], list[str]]]: the extras of the packages
# keyed by the markers that need to be true.
extras = {
name: {
markers: sorted(pkg_extras)
for markers, pkg_extras in sorted(by_markers.items())
}
for name, by_markers in sorted(extras.items())
},
)

def _add_marker(markers, marker):
if not marker or marker in markers:
return markers
return tuple(sorted(markers + (marker,)))

def _is_processed(processed, markers):
"""Tell if a group was already processed with the same or looser markers.

Args:
processed: {type}`list[tuple[str]]` the markers that the group was
processed with.
markers: {type}`tuple[str]` the markers to check.

Returns:
{type}`bool` `True` if all of the markers of a processed entry are in
`markers`, i.e. the requirements of the group are already requested
whenever `markers` are true.
"""
for known in processed:
if len([m for m in known if m in markers]) == len(known):
return True
return False

def _match(spec, groups):
"""Match a dependency group specifier against normalized group keys.

Expand Down
Loading
Loading