Repository navigation
Conversation
Merging this PR will regress 4 benchmarks4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
4917279 to
2623443
Compare
Walltime benchmarks run many rounds, but the results only carried summary stats. Report the distribution as well: - Each benchmark's stats gain `percentiles_ns`, p1 to p99 of the time per iteration. They use statrs' quantile formula on the rounds sorted once: statrs runs an O(n) selection per call, so 99 calls took 5.2 s on 10M rounds against 0.38 s for one sort. q1, median and q3 now come from them: q1 and q3 are unchanged, and the median, mean and stdev can differ in the last floating-point bit. - cargo-codspeed writes the time and iteration count of every round, in execution order, to `walltime_rounds/<pid>.json.gz` next to `results/<pid>.json`. Above 100,000 rounds, a benchmark keeps a uniformly random subsample, still in execution order, and then has fewer rounds than `stats.rounds`: on a 9.7M-round benchmark, that subsample kept p1 to p99 within 1.5% and the CDF within 0.4%. The runner uploads walltime profile folders uncompressed, so the file is gzipped, at the fastest level: it runs 11 times faster than the default on 2.5M rounds, for a file 20 to 30% larger. The per-benchmark files under `target/codspeed/walltime/raw_results/` now carry the rounds until cargo-codspeed aggregates them, so they are written as compact JSON and read through a buffer. Results files keep parsing with older consumers: the new stats field is optional, and the rounds live in their own file. Refs COD-4241
2623443 to
1eece48
Compare
Report the distribution of walltime measurements, not only their summary stats.
results/<pid>.jsongainpercentiles_ns, p1 to p99 of the time per iteration. They use statrs' quantile formula on the rounds sorted once: statrs runs an O(n) selection per call, so 99 calls took 5.2 s on 10M rounds against 0.38 s for one sort.q1_ns,median_nsandq3_nsnow come from the same list: q1 and q3 are unchanged, and the median, mean and stdev can differ in the last floating-point bit.cargo-codspeedwrites the time and iteration count of every round, in execution order, towalltime_rounds/<pid>.json.gzin the profile folder:{"benchmarks": [{"uri": "...::sort", "times_per_round_ns": [50649, 40082, ...], "iters_per_round": [1, 1, ...]}]}times_per_round_nsthanstats.rounds. On a 9.7M-round benchmark, a 100k subsample kept p1 to p99 within 1.5% and the CDF within 0.4%. The file is gzipped because the runner uploads walltime profile folders uncompressed: on a 100k-round Go benchmark, gzip took the same format from 802 KB to 231 KB. It runs at its fastest level, which compresses 2.5M rounds 11 times faster than the default level, for a file 20 to 30% larger.The intermediate per-benchmark files under
target/codspeed/walltime/raw_results/carry the rounds untilcargo-codspeedaggregates them, so they are now written as compact JSON and read through a buffer.Compatibility:
percentiles_nsis optional, and matches the fieldrunner-sharedaccepts in CodSpeedHQ/codspeed#630. The rounds live in their own file, so current consumers of the results file are unaffected. Per-benchmark files from older compat crates still load.CodSpeedHQ/codspeed-go#70 copies the percentile and sampling code into go-runner; a
TODO(COD-4241)in both asks to share one implementation.Refs COD-4241