Skip to content

fix(criterion): document the async API on docs.rs - #207

Open
not-matthias wants to merge 3 commits into
mainfrom
cod-4302-criterion-compat-docsrs-async
Open

not-matthias wants to merge 3 commits into
mainfrom
cod-4302-criterion-compat-docsrs-async

Conversation

@not-matthias

@not-matthias not-matthias commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Fixes COD-4302

Bencher::to_async was missing from the codspeed-criterion-compat docs (reported in #6). The method still exists, but it is gated behind the async feature, and docs.rs built the crate with default features only.

  • Add the same [package.metadata.docs.rs] section as upstream Criterion, so docs.rs builds with all async runtimes enabled (async_futures, async_smol, async_std, async_tokio).
  • Enable doc_cfg under docsrs, so feature-gated items get an "Available on crate feature async only" badge. The internal codspeed cfg is hidden from the badges; it only selects between the two compat_criterion modules and would otherwise mark every item as "non-codspeed only".
  • Bump the walltime fork to smol 2 and update async-std to 1.13 in the lockfile. Both previously pulled in async-io 1 → rustix 0.37, which no longer compiles on nightly (rustc_* attributes are reserved), so the async docs build would fail on docs.rs. This also fixes the fork's own docs.rs build, broken since 5.0.0. smol is not part of the public API (SmolExecutor only calls smol::block_on), so this is not a breaking change.

Verified locally:

  • RUSTDOCFLAGS="--cfg docsrs" cargo +nightly doc -p codspeed-criterion-compat -p codspeed-criterion-compat-walltime --no-deps with all async features passes (failed before the bump), and Bencher shows to_async with the async feature badge.
  • cargo test -p codspeed-criterion-compat-walltime --lib with all async features passes.

- `Bencher::to_async` is gated behind the `async` feature, and docs.rs built
  the crate with default features only, so the async API was missing from
  the published docs.
- Enable all async runtimes for the docs.rs build, like upstream Criterion.
@not-matthias
not-matthias marked this pull request as ready for review November 17, 2026 17:35
@greptile-apps

greptile-apps Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] This PR appears safe to merge.

Summary

Adds docs.rs settings so codspeed-criterion-compat documents its async API.

  • The docs.rs API page now includes the async bencher method.

Reviews (1) · Last reviewed commit: "fix(criterion): document the async API o..." · Reviewed by Greptile

@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.07%

2 warnings, 1 note

⚠️ Unable to measure the execution of 59 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

ℹ️ 57 benchmarks have no flamegraph

  • The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze. (9 benchmarks)
  • The flamegraph wasn't generated: no execution data was recorded while the benchmark ran. Make sure the compiler didn't optimize the benchmark out. (48 benchmarks)

See affected benchmarks

⚡ 6 improved benchmarks
❌ 8 regressed benchmarks
✅ 604 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime hamiltonian_cycle[5] 905 ns 1,523 ns -40.58%
❌ Walltime mul 2 ns 3 ns -33.33%
❌ Walltime rem 5 ns 6 ns -16.67%
❌ Walltime instant 37 ns 41 ns -9.76%
❌ Walltime n_queens_solver[4] 2.1 µs 2.3 µs -6.35%
❌ Walltime iterative[10] 17 ns 18 ns -5.56%
❌ Walltime init_array[42] 35 ns 37 ns -5.41%
❌ Walltime zeroed_buffer[1] 48.9 µs 50.4 µs -3.03%
⚡ Walltime find_highest_set_bit[0] 7 ns 5 ns +40%
⚡ Walltime spawn_on_parent_runtime 25.5 µs 19.5 µs +30.2%
⚡ Walltime recursive_memoized[HashMap<u64, u64>, 10] 2.1 µs 1.8 µs +16.29%
⚡ Walltime bench_array2[42] 41 ns 36 ns +13.89%
⚡ Walltime add_two_integers[(255, 255)] 20 ns 19 ns +5.26%
⚡ Walltime iter_with_setup 40 ns 38 ns +5.26%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-4302-criterion-compat-docsrs-async (7dd2d13) with main (1e075ad)

Open in CodSpeed

@svix-jplatte

Copy link
Copy Markdown

I would recommend you also add #![cfg_attr(docsrs, feature(doc_cfg))] to the top of lib.rs, so the feature requirements show up on docs.rs.

- Enable `doc_cfg` under `docsrs` so rustdoc labels feature-gated items,
  e.g. `Bencher::to_async` as "Available on crate feature `async` only".
- Hide the internal `codspeed` cfg from the labels: it selects between the
  two `compat_criterion` modules, and would otherwise mark every item as
  "Available on non-codspeed only".
Comment thread crates/criterion_compat/Cargo.toml Resolved
- docs.rs builds with nightly, and `rustix` 0.37 no longer compiles there
  (`rustc_*` attributes are reserved). Both async docs builds pulled it in:
  `smol` 1 and the locked `async-std` 1.12 depend on `async-io` 1.
- Bump the walltime fork to `smol` 2, as the compat crate already uses, and
  update `async-std` to 1.13 in the lockfile. Both now use `async-io` 2.
- `smol` is not part of the public API: `SmolExecutor` only calls
  `smol::block_on`.

@lvaroqui lvaroqui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm 👍

Comment on lines +1 to 2
#![cfg_attr(docsrs, feature(doc_cfg), doc(auto_cfg(hide(codspeed))))]
pub use codspeed::codspeed_uri;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nit: I would add a new line between the crate level attr and the first pub use to not misread it as an outer attribute of pub use codspeed::codspeed_uri; (since we have one on pub use codspeed::telemetry;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants