Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions docs/pypi/use.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,17 @@ platforms, for example `foo[bar]` on Linux and `foo[baz]` on macOS, then
`@pypi//foo:[bar]` on macOS is the same as `@pypi//foo:[]`.

The dependencies of the packages use the extras written in the wheel
`METADATA`, so if `foo` depends on `bar[socks]`, then `@pypi//foo` depends on
`@pypi//bar:[socks]`. If the extra is not in the lock file, then building
`@pypi//foo` fails with a message asking you to add `bar[socks]` to the
requirements or the lock file. For example, when compiling the requirements
with `uv pip compile`, pass `--no-strip-extras` to keep the extras in the
output.
`METADATA`, so if `foo` depends on `bar[socks]`, then `@pypi//foo` includes
`bar` and the dependencies of its `socks` extra. The extras of the
dependencies are resolved by the repository of `bar` from its own `METADATA`,
so they do not need to be in the lock file, e.g. when the requirements are
compiled with the extras stripped. This does not change `@pypi//bar`, which
only includes the extras requested in the lock file. The dependencies of the
extra that are not in the lock file are left out like any other dependency
that is not in the lock file. If `bar` does not provide the `socks` extra,
then only `bar` is used and a warning is printed when `@pypi//foo` is
analyzed, e.g. `WARNING: bar 1.0 does not provide the extra 'socks'`, which is
how `pip` handles the unknown extras.

If the lock file requests an extra that the package does not provide, e.g.
`foo[nope]`, then `@pypi//foo:[nope]` is the same as `@pypi//foo:[]`, because
Expand All @@ -142,9 +147,7 @@ When the `requirements.bzl` file is vendored, there is no hub repo and the
labels returned by the `requirement` helper point to the repositories of the
wheels. Their `pkg` targets include the extras from the lock file and they
have the `[]` target and the `[extra]` targets for the extras used in the lock
file and for all of the extras that the package provides. The dependencies
between the packages use the `pkg` targets and ignore the extras from the wheel
`METADATA` like before, so the extras need to be in the lock file.
file and for all of the extras that the package provides.

The `whl` targets, e.g. `@pypi//foo:whl`, do not use the extras and include
only the wheels of the package and its dependencies without extras.
Expand Down
25 changes: 12 additions & 13 deletions news/2786.changed.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
(pypi) The dependencies between the PyPI packages now use the extras from the
wheel `METADATA`, e.g. if `foo` requires `bar[socks]`, then `@pypi//foo`
depends on `@pypi//bar:[socks]` instead of `@pypi//bar`. Previously such extras
were ignored. If the lock file of a hub does not request the extra, the build
fails with a message asking to add it, e.g. by compiling the requirements with
`--no-strip-extras`. This does not apply when there is no hub, e.g. with a
vendored `requirements.bzl`, where the dependencies use the `pkg` targets,
which include the extras from the lock file, as before. The `whl` targets no longer include the wheels of the
dependencies of the extras. A member of `experimental_requirement_cycles`
without extras, e.g. `foo`, is now only the package without extras, so the
cycles that go through `foo[bar]` need `foo[bar]` as a member. The same
distribution that is requested with different extras on different platforms
now uses a single whl repository. The `@pypi//foo` targets still include the
extras requested in the lock file. An extra that the package does not provide
is still ignored, but a warning is now printed like `pip` does, e.g.
includes `bar` and the dependencies of its `socks` extra. Previously such
extras were ignored. The extra does not need to be requested in the lock file
of the hub, because it is resolved by the whl repository of `bar`, and the
dependencies of the extra that are not in the lock file are left out. The
`whl` targets no longer include the wheels of the dependencies of the extras.
A member of `experimental_requirement_cycles` without extras, e.g. `foo`, is
now only the package without extras, so the cycles that go through `foo[bar]`
need `foo[bar]` as a member. The same distribution that is requested with
different extras on different platforms now uses a single whl repository. The
`@pypi//foo` targets still include the extras requested in the lock file. An
extra that the package does not provide is still ignored, but a warning is now
printed like `pip` does, e.g.
`WARNING: foo 1.0 does not provide the extra 'nope'`
([#2786](https://github.com/bazel-contrib/rules_python/issues/2786)).
3 changes: 0 additions & 3 deletions python/private/internal_dev_deps.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,6 @@ def _internal_dev_deps_impl(mctx):
name = "whl_library_extras_pkg",
root = "//tests/pypi/whl_library/testdata/pkg:BUILD.bazel",
output = "pkg-1.0-any-none-any.whl",
# Without a hub the dependencies use the `pkg` target, which includes
# the extras of the requirement.
requirement = "pkg[optional]",
# The following is necessary to enable pipstar and make tests faster
config_load = "@rules_python//tests/pypi/whl_library/testdata:packages.bzl",
dep_template = "@whl_library_extras_{name}//:{target}",
Expand Down
3 changes: 2 additions & 1 deletion python/private/pypi/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,7 @@ bzl_library(
name = "pkg_aliases",
srcs = ["pkg_aliases.bzl"],
deps = [
":extra_dep",
":labels",
":whl_library_deps_targets",
"//python:py_library",
Expand Down Expand Up @@ -643,8 +644,8 @@ bzl_library(
srcs = ["whl_library_deps_targets.bzl"],
deps = [
":env_marker_setting",
":extra_dep",
":labels",
":missing_package",
":pep508_deps",
"//python:py_library",
"//python/private:normalize_name",
Expand Down
5 changes: 3 additions & 2 deletions python/private/pypi/hub_repository.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,9 @@ def _impl(rctx):
"config.bzl",
rctx.attr._config_template,
substitutions = {
# The available extras of each package for the spoke repositories
# to check if the extras of their dependencies are available.
# The requested extras of each package for the spoke repositories
# to create the targets of the extras that the package does not
# provide.
"%%PACKAGES%%": render.dict(available_extras(whl_map), value_repr = render.list),
},
)
Expand Down
52 changes: 13 additions & 39 deletions python/private/pypi/missing_package.bzl
Original file line number Diff line number Diff line change
@@ -1,16 +1,25 @@
"""Rules for generating an execution-phase action failure when a PyPI package or its extra is missing."""
"""Rule for generating an execution-phase action failure when a PyPI package is missing."""

load("//python/private:py_info.bzl", "PyInfo")
load("//python/private:reexports.bzl", "BuiltinPyInfo")

def _error(ctx, message):
def _missing_package_error_impl(ctx):
out = ctx.actions.declare_file(ctx.label.name + ".error")

if ctx.attr.hub_name:
hub_clause = ' when building under PyPI hub "{hub}". Try adding it to the requirements of this hub (e.g. requirements_lock or requirements_by_platform in pip.parse)'.format(
hub = ctx.attr.hub_name,
)
else:
hub_clause = ' because no default PyPI hub was configured. Try designating a default hub via pip.default(default_hub = "...") or select a hub using --@rules_python//python/config_settings:venv'

# Register an action that fails when Bazel attempts to stage/build this file
ctx.actions.run_shell(
outputs = [out],
command = "echo \"$1\" >&2 && exit 1",
arguments = ["ERROR: " + message],
command = "echo 'ERROR: PyPI package \"{pkg}\" is not available{hub_clause}.' >&2 && exit 1".format(
pkg = ctx.attr.package_name,
hub_clause = hub_clause,
),
)

maybe_builtin = [BuiltinPyInfo(transitive_sources = depset([out]))] if BuiltinPyInfo != None else []
Expand All @@ -25,45 +34,10 @@ def _error(ctx, message):
),
] + maybe_builtin

def _missing_package_error_impl(ctx):
if ctx.attr.hub_name:
hub_clause = ' when building under PyPI hub "{hub}". Try adding it to the requirements of this hub (e.g. requirements_lock or requirements_by_platform in pip.parse)'.format(
hub = ctx.attr.hub_name,
)
else:
hub_clause = ' because no default PyPI hub was configured. Try designating a default hub via pip.default(default_hub = "...") or select a hub using --@rules_python//python/config_settings:venv'

return _error(ctx, 'PyPI package "{pkg}" is not available{hub_clause}.'.format(
pkg = ctx.attr.package_name,
hub_clause = hub_clause,
))

missing_package_error = rule(
implementation = _missing_package_error_impl,
attrs = {
"hub_name": attr.string(mandatory = True),
"package_name": attr.string(mandatory = True),
},
)

def _missing_extra_error_impl(ctx):
return _error(ctx, (
'The PyPI package "{pkg}" requires "{dep}[{extra}]", but the extra "{extra}" of ' +
'"{dep}" is not available in the hub. Add "{dep}[{extra}]" to the requirements or ' +
"the lock file of the hub, e.g. by compiling the requirements with the extras kept " +
"(`--no-strip-extras`) or by adding the extra to the project dependencies."
).format(
pkg = ctx.attr.required_by,
dep = ctx.attr.package_name,
extra = ctx.attr.extra,
))

missing_extra_error = rule(
implementation = _missing_extra_error_impl,
doc = "A target that fails the build with a helpful message when an extra of a PyPI package is missing.",
attrs = {
"extra": attr.string(mandatory = True),
"package_name": attr.string(mandatory = True),
"required_by": attr.string(mandatory = True),
},
)
6 changes: 3 additions & 3 deletions python/private/pypi/pip_repository.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -286,9 +286,9 @@ def _pip_repository_impl(rctx):
"config.bzl",
rctx.attr._config_template,
substitutions = {
# The available extras of each package for the spoke
# repositories to check if the extras of their dependencies
# are available.
# The requested extras of each package for the spoke
# repositories to create the targets of the extras that the
# package does not provide.
"%%PACKAGES%%": render.dict({
pkg: sorted(selected_extras.get(pkg, []))
for pkg in bzl_packages
Expand Down
12 changes: 11 additions & 1 deletion python/private/pypi/pkg_aliases.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ load("@bazel_skylib//lib:selects.bzl", "selects")
load("//python:py_library.bzl", "py_library")
load("//python/private:common_labels.bzl", "labels")
load("//python/private:text_util.bzl", "render")
load(":extra_dep.bzl", "EXTRAS_SELECT_TARGET", "extra_dep", "extra_select")
load(
":labels.bzl",
"DATA_LABEL",
Expand Down Expand Up @@ -124,6 +125,8 @@ def pkg_aliases(

_native = kwargs.pop("native", native)
rules = kwargs.pop("rules", struct(
extra_dep = extra_dep,
extra_select = extra_select,
py_library = py_library,
whl_library_deps_targets = whl_library_deps_targets,
))
Expand Down Expand Up @@ -157,6 +160,10 @@ def pkg_aliases(
# targets do not use the extras, so only the package without extras
# can be a member of the group.
target_names = {
# The whl repositories resolve the extras requested by the
# packages that depend on them, so the hub only needs to forward
# to the whl repository of the target platform.
EXTRAS_SELECT_TARGET: EXTRAS_SELECT_TARGET,
WHEEL_FILE_PUBLIC_LABEL: WHEEL_FILE_IMPL_LABEL if "" in group_extras else WHEEL_FILE_PUBLIC_LABEL,
}

Expand Down Expand Up @@ -214,7 +221,10 @@ def pkg_aliases(
fail("The `actual` arg must be a dictionary or a string")

kwargs = {}
if target_name.startswith("_"):
if target_name == EXTRAS_SELECT_TARGET:
# Used by the whl repositories that depend on the extras.
pass
elif target_name.startswith("_"):
kwargs["visibility"] = ["//_groups:__subpackages__"]
elif target_name in (WHEEL_FILE, PY_SRCS_LABEL):
kwargs["visibility"] = ["//visibility:private"]
Expand Down
2 changes: 1 addition & 1 deletion python/private/pypi/render_pkg_aliases.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ def _non_versioned_platform(p, *, strict = False):
fail("Expected to always have a platform in the form '{{abi}}_{{os}}_{{arch}}', got: {}".format(p))

def available_extras(aliases):
"""Return the extras that are available in the hub for each package.
"""Return the extras that are requested in the hub for each package.

Args:
aliases: {type}`dict[str, dict[struct, str]]` the packages to the
Expand Down
Loading
Loading