Skip to content

feat: report walltime percentiles and rounds - #205

Draft
fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles-and-rounds
Draft

fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles-and-rounds

Conversation

@fargito

@fargito fargito commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Report the distribution of walltime measurements, not only their summary stats.

  • Percentiles: each benchmark's stats in results/<pid>.json gain percentiles_ns, p1 to p99 of the time per iteration. They use statrs' quantile formula on the rounds sorted once: statrs runs an O(n) selection per call, so 99 calls took 5.2 s on 10M rounds against 0.38 s for one sort. q1_ns, median_ns and q3_ns now come from the same list: q1 and q3 are unchanged, and the median, mean and stdev can differ in the last floating-point bit.
  • Rounds: cargo-codspeed writes the time and iteration count of every round, in execution order, to walltime_rounds/<pid>.json.gz in the profile folder:
    {"benchmarks": [{"uri": "...::sort", "times_per_round_ns": [50649, 40082, ...], "iters_per_round": [1, 1, ...]}]}
    Above 100,000 rounds, a benchmark keeps a uniformly random subsample, still in execution order, and then has fewer times_per_round_ns than stats.rounds. On a 9.7M-round benchmark, a 100k subsample kept p1 to p99 within 1.5% and the CDF within 0.4%. The file is gzipped because the runner uploads walltime profile folders uncompressed: on a 100k-round Go benchmark, gzip took the same format from 802 KB to 231 KB. It runs at its fastest level, which compresses 2.5M rounds 11 times faster than the default level, for a file 20 to 30% larger.

The intermediate per-benchmark files under target/codspeed/walltime/raw_results/ carry the rounds until cargo-codspeed aggregates them, so they are now written as compact JSON and read through a buffer.

Compatibility: percentiles_ns is optional, and matches the field runner-shared accepts in CodSpeedHQ/codspeed#630. The rounds live in their own file, so current consumers of the results file are unaffected. Per-benchmark files from older compat crates still load.

CodSpeedHQ/codspeed-go#70 copies the percentile and sampling code into go-runner; a TODO(COD-4241) in both asks to share one implementation.

Refs COD-4241

@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 4 benchmarks

4 warnings, 1 note

⚠️ Unable to measure the execution of 59 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran on different hardware

Its base and head results were measured on machines with a different CPU, core count or memory, so the change might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 4 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

ℹ️ 57 benchmarks have no flamegraph

  • The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze. (9 benchmarks)
  • The flamegraph wasn't generated: no execution data was recorded while the benchmark ran. Make sure the compiler didn't optimize the benchmark out. (48 benchmarks)

See affected benchmarks

⚡ 4 improved benchmarks
❌ 4 regressed benchmarks
✅ 610 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime add 2 ns 3 ns -33.33%
❌ Walltime vec_copy_with_bytes_counter 41 ns 58 ns -29.31%
❌ Walltime spawn_on_parent_runtime 21.6 µs 26.4 µs -18.13%
❌ Walltime zeroed_buffer[1] 48.2 µs 50.4 µs -4.45%
⚡ CPU Simulation spawn_on_parent_runtime 70.1 µs 33.4 µs ×2.1
⚡ Walltime find_highest_set_bit[0] 7 ns 5 ns +40%
⚡ Walltime build_vec 237 ns 214 ns +10.75%
⚡ Walltime knight_tour[8] 299.1 ms 294.9 ms +1.41%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/walltime-percentiles-and-rounds (1eece48) with main (0c6d629)

Open in CodSpeed

@fargito fargito changed the title feat: report walltime percentiles and raw rounds feat: report walltime percentiles and rounds Nov 18, 2026
@fargito
fargito force-pushed the feat/walltime-percentiles-and-rounds branch from 4917279 to 2623443 Compare November 18, 2026 14:58
Walltime benchmarks run many rounds, but the results only carried summary
stats. Report the distribution as well:

- Each benchmark's stats gain `percentiles_ns`, p1 to p99 of the time per
  iteration. They use statrs' quantile formula on the rounds sorted once:
  statrs runs an O(n) selection per call, so 99 calls took 5.2 s on 10M
  rounds against 0.38 s for one sort. q1, median and q3 now come from them:
  q1 and q3 are unchanged, and the median, mean and stdev can differ in the
  last floating-point bit.
- cargo-codspeed writes the time and iteration count of every round, in
  execution order, to `walltime_rounds/<pid>.json.gz` next to
  `results/<pid>.json`. Above 100,000 rounds, a benchmark keeps a uniformly
  random subsample, still in execution order, and then has fewer rounds
  than `stats.rounds`: on a 9.7M-round benchmark, that subsample kept p1 to
  p99 within 1.5% and the CDF within 0.4%. The runner uploads walltime
  profile folders uncompressed, so the file is gzipped, at the fastest
  level: it runs 11 times faster than the default on 2.5M rounds, for a
  file 20 to 30% larger.

The per-benchmark files under `target/codspeed/walltime/raw_results/` now
carry the rounds until cargo-codspeed aggregates them, so they are written
as compact JSON and read through a buffer.

Results files keep parsing with older consumers: the new stats field is
optional, and the rounds live in their own file.

Refs COD-4241
@fargito
fargito force-pushed the feat/walltime-percentiles-and-rounds branch from 2623443 to 1eece48 Compare November 18, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant