Repository navigation
Description
Tracking issue for the research on the extras of the PyPI dependencies
that goes beyond bazel-contrib#4490. The goal is to match
what pip and rules_pycross do: an extra on a dependency edge works
without the user requesting it in the lock file.
Upstream issue: bazel-contrib#2786.
Background
- feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 (upstream, draft) adds the
[extra]
targets:@pypi//foo(base + the extras from the lock file),
@pypi//foo:[],@pypi//foo:[x]. An edge likepytest-cov->
coverage[toml]fails there if the lock file does not have
coverage[toml]. - rules_pycross never reads the wheel
METADATA; it uses the edges of
the lock file, wherecoverage[toml]is its own node, so it never
asks whether the user requested the extra. - feat(pypi): resolve the extras of the dependencies in the whl repositories bazel-contrib/rules_python#4523 and feat(pypi): expose the pyproject.toml extras with pip.configure bazel-contrib/rules_python#4522
were closed upstream to reduce the noise and are continued here.
Status
Everything is implemented and split into a stack of draft PRs, each
based on the previous one and the first one on the branch of bazel-contrib#4490.
Each has one commit and bazel test --config=fast-tests //tests/pypi/... passes on each of them (384 tests at the tip). #40
was the combined PR and is closed.
| PR | What | Size |
|---|---|---|
| #42 | extra_dep and extra_select rules, nothing uses them |
5 files, +676 |
| #43 | Edges of the whl repositories use them, missing_extra_error removed |
16 files, +652 -198 |
| #44 | pep508 deps returns the missing dependencies (pure function) |
2 files, +103 -1 |
| #45 | Hub exports the marker environments of the target platforms | 7 files, +73 |
| #46 | Warning about the dependencies that are not in the hub | 11 files, +255 -19 |
| #47 | select_dependency_groups collects the extras (pure function) |
2 files, +180 -15 |
| #48 | pip.configure adds the pyproject.toml extras to the hubs |
8 files, +488 -13 |
#42 to #46 are one topic (edges), #47 and #48 are another
(pyproject.toml extras) and only depend on bazel-contrib#4490, they are in the same
stack because they touch the same files as #45.
-
extra_dep/extra_selectrules: the extra of an edge is
selected in the whl repository of the dependency with a private
build setting, so no label points at a target that may not exist.
Aspects propagate overdepsand all providers are forwarded. - The edges of the whl repositories use them, with and without a
hub.missing_extra_erroris removed. - An extra that the package does not provide is the package without
extras plus a warning, likepip. -
bazel queryandbazel cquerywork across the edges (a
genquerytest with real whl repositories). - Warning about the dependencies that are left out because they are
not in the hub, but whose marker can be true on a target platform. -
pip.configureadds the extras of thepyproject.toml
requirements per target platform. - Split feat(pypi): resolve the extras of the dependencies in the whl repositories and expose the pyproject.toml extras #40 into the stack feat(pypi): add the extra_dep and extra_select rules #42 to feat(pypi): add the extras from the pyproject.toml to the hubs with pip.configure #48.
Not done / known gaps
- Not run:
//docs:docs, the nested Bazel integration tests, a real
WORKSPACEpip_parsebuild. The fork has no Buildkite. - No warnings about the left out dependencies with
pip_parsein
WORKSPACE. - The hub has two similar outputs:
platforms(for thepyproject.toml
extras) andtarget_platform_envs(for the warning). - The cycles through the extras, e.g.
a[x]->b[y]->a[x], are
real dependency cycles now and need
experimental_requirement_cycles. - Only two of the eight edge scenarios run through real whl
repositories, the rest are macro tests with mocks and rule tests.
Blockers
- feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 has to land first, and it waits for
refactor(pypi): add the helpers for the extra targets bazel-contrib/rules_python#4492 and
refactor(pypi): extract the failing action of missing_package_error bazel-contrib/rules_python#4493. All three are drafts without a
review. - The decisions below.
Decisions needed from @aignas
- Is a public
_extrastarget in every hub package and whl repository
acceptable? - Should the cycles through the extras be grouped automatically or is
documentingexperimental_requirement_cyclesenough? - A hub with a single lock file and the default platforms treats all of
the platforms as targets, so a lock file compiled on Linux warns about
colorama; sys_platform == "win32". Keep, make it opt-out, or lower
the level? - Should the config settings of a wheel be split per platform when the
pyproject.tomlextras differ per platform? A platform outside the
target_platformsof the hub then no longer matches the package. - Which parts go upstream after feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 lands, and in which order?
Next steps for @aignas
- Get refactor(pypi): extract the failing action of missing_package_error bazel-contrib/rules_python#4493 and refactor(pypi): add the helpers for the extra targets bazel-contrib/rules_python#4492 reviewed and merged upstream, then mark feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490
ready. - Review the stack feat(pypi): add the extra_dep and extra_select rules #42 to feat(pypi): add the extras from the pyproject.toml to the hubs with pip.configure #48 in order and answer the decisions above.
- After feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 lands: the stack is merged with
mainand proposed
upstream one PR at a time.
Activity