Repository navigation
feat: export the telemetry of criterion and bencher benchmarks over OTLP - #204
Conversation
Merging this PR will degrade performance by 5.8%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
[Medium impact] The PR appears safe to merge. SummaryAdds optional OTLP export for Criterion benchmarks through the existing
No actionable issues found. Review used source inspection; no builds or runtime tests were run.
|
Add a `telemetry` feature to `codspeed-criterion-compat`, as `codspeed-divan-compat` has. When `CODSPEED_OTEL_ENDPOINT` is set, the `main` of `criterion_main!` installs the `tracing` subscriber of `codspeed`, which exports spans as traces, and `tracing` events and `log` records as logs, and keeps it until the groups are done. The export is the one of the `codspeed` crate, so only the telemetry build of `cargo-codspeed` compiles it. The macro expands in the user's crate, where it can't check the feature of the compat crate, so it calls a hidden function that is a no-op without the feature. `criterion::telemetry` re-exports the module, for custom `main` functions and benchmarks installing their own subscriber. `benches/telemetry.rs` emits telemetry in each way the divan example does. It needs the feature, so builds without it skip it. Closes COD-4287 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add the `telemetry` feature to `codspeed-bencher-compat` too, wired as in `codspeed-criterion-compat`: the `main` of `benchmark_main!` installs the `tracing` subscriber of `codspeed` when `CODSPEED_OTEL_ENDPOINT` is set, through a hidden function that is a no-op without the feature, and `bencher::telemetry` re-exports the module. `benches/telemetry.rs` emits telemetry in each way the other examples do, behind the feature. Closes COD-4287 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The standalone macOS job pinned the runner to `branch:main`. The action installs from a branch by sourcing `~/.cargo/env`, which does not exist on `macos-latest`, so the job failed before running any benchmark. The released runner now supports macOS, so drop the pin and run the job as one more entry of the walltime integration matrix. Build with `-m walltime` for every entry: `macos-latest` does not set `CODSPEED_RUNNER_MODE`, so the build would otherwise use analysis mode. Skip the upload on macOS, as the old job did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dedfc01 to
1e075ad
Compare
Export the traces and logs that Criterion.rs and bencher benchmarks emit with
tracingandlog, as #197 did for divan. Using it takes thetelemetryfeature ofcodspeed-criterion-compatorcodspeed-bencher-compat, and a telemetry build:cargo codspeed build -m telemetry # or with other modes, e.g. -m simulation,telemetry codspeed run -m telemetry -- cargo codspeed runWhen
CODSPEED_OTEL_ENDPOINTis set, themainofcriterion_main!orbenchmark_main!installs thetracingsubscriber ofcodspeed, which exports spans as traces, andtracingevents andlogrecords as logs, and keeps it until the groups are done. The export, its filter (CODSPEED_TELEMETRY_FILTER) and its timing attributes are the ones ofcodspeed, unchanged: only the telemetry build ofcargo-codspeedcompiles them.One commit per layer: criterion first, bencher second with the same wiring.
Worth a careful look
#[cfg(feature = "telemetry")]would read the user's features, not ours. So each macro calls a hidden__init_telemetry()of its compat crate, which returns the guard with the feature andNonewithout it.bencher, used outside ofcargo codspeed, are unchanged.main(theCriterion::new_instrumented()pattern) callscriterion::telemetry::init()itself and keeps the guard, as a divan benchmark with its own subscriber addsdivan::telemetry::layer(). This is for the docs, along with lifting their divan-only note.crates/criterion_compat/benches/telemetry.rsandcrates/bencher_compat/benches/telemetry.rsemit telemetry in each of the ways the divan example does. They need the feature (required-features), socargo codspeed build -p <layer>without--features telemetry, as in CI, skips them.Checked end to end with a telemetry build of each bench run against a local OTLP/HTTP receiver: the
infospans (parse_and_sum,parse,sum) arrive on/v1/traces, the events andlogrecords on/v1/logs, and thedebugones are filtered out. WithoutCODSPEED_OTEL_ENDPOINT, nothing is sent.Closes COD-4287