Skip to content

Extras of the dependencies without the lock file requesting them #41

Open

Description

@aignas

Tracking issue for the research on the extras of the PyPI dependencies
that goes beyond bazel-contrib#4490. The goal is to match
what pip and rules_pycross do: an extra on a dependency edge works
without the user requesting it in the lock file.

Upstream issue: bazel-contrib#2786.

Background

Status

Everything is implemented and split into a stack of draft PRs, each
based on the previous one and the first one on the branch of bazel-contrib#4490.
Each has one commit and bazel test --config=fast-tests //tests/pypi/... passes on each of them (384 tests at the tip). #40
was the combined PR and is closed.

PR What Size
#42 extra_dep and extra_select rules, nothing uses them 5 files, +676
#43 Edges of the whl repositories use them, missing_extra_error removed 16 files, +652 -198
#44 pep508 deps returns the missing dependencies (pure function) 2 files, +103 -1
#45 Hub exports the marker environments of the target platforms 7 files, +73
#46 Warning about the dependencies that are not in the hub 11 files, +255 -19
#47 select_dependency_groups collects the extras (pure function) 2 files, +180 -15
#48 pip.configure adds the pyproject.toml extras to the hubs 8 files, +488 -13

#42 to #46 are one topic (edges), #47 and #48 are another
(pyproject.toml extras) and only depend on bazel-contrib#4490, they are in the same
stack because they touch the same files as #45.

Not done / known gaps

  • Not run: //docs:docs, the nested Bazel integration tests, a real
    WORKSPACE pip_parse build. The fork has no Buildkite.
  • No warnings about the left out dependencies with pip_parse in
    WORKSPACE.
  • The hub has two similar outputs: platforms (for the pyproject.toml
    extras) and target_platform_envs (for the warning).
  • The cycles through the extras, e.g. a[x] -> b[y] -> a[x], are
    real dependency cycles now and need
    experimental_requirement_cycles.
  • Only two of the eight edge scenarios run through real whl
    repositories, the rest are macro tests with mocks and rule tests.

Blockers

  1. feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 has to land first, and it waits for
    refactor(pypi): add the helpers for the extra targets bazel-contrib/rules_python#4492 and
    refactor(pypi): extract the failing action of missing_package_error bazel-contrib/rules_python#4493. All three are drafts without a
    review.
  2. The decisions below.

Decisions needed from @aignas

  1. Is a public _extras target in every hub package and whl repository
    acceptable?
  2. Should the cycles through the extras be grouped automatically or is
    documenting experimental_requirement_cycles enough?
  3. A hub with a single lock file and the default platforms treats all of
    the platforms as targets, so a lock file compiled on Linux warns about
    colorama; sys_platform == "win32". Keep, make it opt-out, or lower
    the level?
  4. Should the config settings of a wheel be split per platform when the
    pyproject.toml extras differ per platform? A platform outside the
    target_platforms of the hub then no longer matches the package.
  5. Which parts go upstream after feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 lands, and in which order?

Next steps for @aignas

  1. Get refactor(pypi): extract the failing action of missing_package_error bazel-contrib/rules_python#4493 and refactor(pypi): add the helpers for the extra targets bazel-contrib/rules_python#4492 reviewed and merged upstream, then mark feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490
    ready.
  2. Review the stack feat(pypi): add the extra_dep and extra_select rules #42 to feat(pypi): add the extras from the pyproject.toml to the hubs with pip.configure #48 in order and answer the decisions above.
  3. After feat(pypi): add targets for the extras of the packages bazel-contrib/rules_python#4490 lands: the stack is merged with main and proposed
    upstream one PR at a time.

Activity

  1. self-assigned this
    on Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions