Skip to content

feat(runner-shared): accept percentiles in walltime benchmark stats - #630

Draft
fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles
Draft

fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles

Conversation

@fargito

@fargito fargito commented Nov 17, 2026

Copy link
Copy Markdown
Member

Add an optional percentiles_ns field to the walltime BenchmarkStats: p1 to p99 of the time per iteration, in nanoseconds like the other _ns stats.

Integrations already hold every round in memory but only report min, max, mean, stdev and quartiles. Reporting the full percentile set lets the backend keep the shape of each benchmark's distribution.

The field is optional and skipped when absent, so results files from current integrations keep parsing, and exec-harness keeps writing the same JSON.

Refs COD-4241

@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 53 untouched benchmarks
⏩ 6 skipped benchmarks1


Comparing feat/walltime-percentiles (76030c0) with main (ffdafa6)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Integrations can report p1 to p99 of the time per iteration in
`stats.percentiles_ns`. The field is optional: results files without it
keep parsing, and it is left out of the JSON when absent.

Refs COD-4241
@fargito
fargito force-pushed the feat/walltime-percentiles branch from 322d73b to 76030c0 Compare November 18, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant