Skip to content

feat: export the telemetry of criterion and bencher benchmarks over OTLP - #204

Merged
lvaroqui merged 3 commits into
mainfrom
cod-4287-telemetry-export-in-codspeed-criterion-compat
Nov 17, 2026
Merged

lvaroqui merged 3 commits into
mainfrom
cod-4287-telemetry-export-in-codspeed-criterion-compat

Conversation

@lvaroqui

@lvaroqui lvaroqui commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Export the traces and logs that Criterion.rs and bencher benchmarks emit with tracing and log, as #197 did for divan. Using it takes the telemetry feature of codspeed-criterion-compat or codspeed-bencher-compat, and a telemetry build:

cargo codspeed build -m telemetry            # or with other modes, e.g. -m simulation,telemetry
codspeed run -m telemetry -- cargo codspeed run

When CODSPEED_OTEL_ENDPOINT is set, the main of criterion_main! or benchmark_main! installs the tracing subscriber of codspeed, which exports spans as traces, and tracing events and log records as logs, and keeps it until the groups are done. The export, its filter (CODSPEED_TELEMETRY_FILTER) and its timing attributes are the ones of codspeed, unchanged: only the telemetry build of cargo-codspeed compiles them.

One commit per layer: criterion first, bencher second with the same wiring.

Worth a careful look

  • The macros expand in the user's crate, where #[cfg(feature = "telemetry")] would read the user's features, not ours. So each macro calls a hidden __init_telemetry() of its compat crate, which returns the guard with the feature and None without it.
  • The macros of the Criterion fork and of bencher, used outside of cargo codspeed, are unchanged.
  • A custom main (the Criterion::new_instrumented() pattern) calls criterion::telemetry::init() itself and keeps the guard, as a divan benchmark with its own subscriber adds divan::telemetry::layer(). This is for the docs, along with lifting their divan-only note.

crates/criterion_compat/benches/telemetry.rs and crates/bencher_compat/benches/telemetry.rs emit telemetry in each of the ways the divan example does. They need the feature (required-features), so cargo codspeed build -p <layer> without --features telemetry, as in CI, skips them.

Checked end to end with a telemetry build of each bench run against a local OTLP/HTTP receiver: the info spans (parse_and_sum, parse, sum) arrive on /v1/traces, the events and log records on /v1/logs, and the debug ones are filtered out. Without CODSPEED_OTEL_ENDPOINT, nothing is sent.

Closes COD-4287

@codspeed

codspeed Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 5.8%

4 warnings, 1 note

⚠️ Unable to measure the execution of 59 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran on different hardware

Its base and head results were measured on machines with a different CPU, core count or memory, so the change might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

ℹ️ 57 benchmarks have no flamegraph

  • The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze. (9 benchmarks)
  • The flamegraph wasn't generated: no execution data was recorded while the benchmark ran. Make sure the compiler didn't optimize the benchmark out. (48 benchmarks)

See affected benchmarks

⚡ 7 improved benchmarks
❌ 10 (👁 5) regressed benchmarks
✅ 601 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime recursive[5] 35 ns 77 ns -54.55%
❌ Walltime iter_batched_large_input 8 ns 11 ns -27.27%
❌ Walltime parse_and_sum 275 ns 296 ns -7.09%
❌ Walltime add_two_integers[(255, 255)] 19 ns 20 ns -5%
❌ Walltime add_two_integers[(42, 13)] 19 ns 20 ns -5%
⚡ Walltime hamiltonian_cycle[4] 1,295 ns 806 ns +60.67%
⚡ Walltime bench_array1[10] 6 ns 5 ns +20%
⚡ Walltime graph_coloring[5] 1.2 µs 1.1 µs +13.08%
⚡ Walltime recursive_memoized[HashMap<u64, u64>, 10] 2.1 µs 1.8 µs +12.14%
👁 Walltime rem 5 ns 6 ns -16.67%
👁 Walltime iterative[10] 17 ns 18 ns -5.56%
👁 Walltime iter_batched_per_iteration 48 ns 52 ns -7.69%
👁 Walltime iter_batched_ref_large_input 5 ns 9 ns -44.44%
👁 Walltime iter_with_setup 42 ns 45 ns -6.67%
👁 Walltime from_elem_decimal[1024] 201 ns 187 ns +7.49%
👁 Walltime spawn_on_parent_runtime 24.5 µs 20.1 µs +21.73%
👁 CPU Simulation spawn_on_parent_runtime 36.2 µs 33.3 µs +8.47%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-4287-telemetry-export-in-codspeed-criterion-compat (1e075ad) with main (aa1bc87)

Open in CodSpeed

@lvaroqui
lvaroqui marked this pull request as ready for review November 17, 2026 09:43
@greptile-apps

greptile-apps Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

Adds optional OTLP export for Criterion benchmarks through the existing codspeed::telemetry API.

  • Criterion benchmark runs can export their telemetry through CodSpeed.

No actionable issues found. Review used source inspection; no builds or runtime tests were run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A["criterion_main!"] --> B["__init_telemetry()"]
  B --> C{"telemetry feature enabled?"}
  C -->|No| D["Return None"]
  C -->|Yes| E["codspeed::telemetry::init()"]
  E --> F["Keep optional guard"]
  D --> G["Run benchmark groups"]
  F --> G
  G --> H["Drop guard and flush exporters"]
Loading

Reviews (1) · Last reviewed commit: "feat(criterion): export the telemetry of..." · Reviewed by Greptile

@lvaroqui lvaroqui changed the title feat(criterion): export the telemetry of benchmarks over OTLP feat: export the telemetry of criterion and bencher benchmarks over OTLP Nov 17, 2026

@not-matthias not-matthias left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

lvaroqui and others added 3 commits November 17, 2026 14:45
Add a `telemetry` feature to `codspeed-criterion-compat`, as
`codspeed-divan-compat` has. When `CODSPEED_OTEL_ENDPOINT` is set, the
`main` of `criterion_main!` installs the `tracing` subscriber of
`codspeed`, which exports spans as traces, and `tracing` events and
`log` records as logs, and keeps it until the groups are done. The
export is the one of the `codspeed` crate, so only the telemetry build
of `cargo-codspeed` compiles it.

The macro expands in the user's crate, where it can't check the feature
of the compat crate, so it calls a hidden function that is a no-op
without the feature. `criterion::telemetry` re-exports the module, for
custom `main` functions and benchmarks installing their own subscriber.

`benches/telemetry.rs` emits telemetry in each way the divan example
does. It needs the feature, so builds without it skip it.

Closes COD-4287
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add the `telemetry` feature to `codspeed-bencher-compat` too, wired as
in `codspeed-criterion-compat`: the `main` of `benchmark_main!` installs
the `tracing` subscriber of `codspeed` when `CODSPEED_OTEL_ENDPOINT` is
set, through a hidden function that is a no-op without the feature, and
`bencher::telemetry` re-exports the module.

`benches/telemetry.rs` emits telemetry in each way the other examples
do, behind the feature.

Closes COD-4287
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The standalone macOS job pinned the runner to `branch:main`. The action
installs from a branch by sourcing `~/.cargo/env`, which does not exist
on `macos-latest`, so the job failed before running any benchmark. The
released runner now supports macOS, so drop the pin and run the job as
one more entry of the walltime integration matrix.

Build with `-m walltime` for every entry: `macos-latest` does not set
`CODSPEED_RUNNER_MODE`, so the build would otherwise use analysis mode.
Skip the upload on macOS, as the old job did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lvaroqui
lvaroqui force-pushed the cod-4287-telemetry-export-in-codspeed-criterion-compat branch from dedfc01 to 1e075ad Compare November 17, 2026 13:45
@lvaroqui
lvaroqui merged commit 1e075ad into main Nov 17, 2026
49 of 50 checks passed
@lvaroqui
lvaroqui deleted the cod-4287-telemetry-export-in-codspeed-criterion-compat branch November 17, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants