Skip to content

fix: clarify benchmark regression report verdicts - #2394

Merged
zeitlinger merged 3 commits into
prometheus:mainfrom
zeitlinger:agent/benchmark-report-verdict
Aug 19, 2026
Merged

fix: clarify benchmark regression report verdicts#2394
zeitlinger merged 3 commits into
prometheus:mainfrom
zeitlinger:agent/benchmark-report-verdict

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

Summary

  • Add an explicit base-vs-head regression verdict using comparable JMH metadata, confidence-interval overlap, and a practical 5% threshold.
  • Remove cross-method Nx slower rankings from the PR-head tables.
  • List head-only benchmarks separately as descriptive results with no regression verdict.

Motivation

The #2329 benchmark report described HistogramBenchmark.openTelemetryExponential as “20x slower” only because it was ranked against a different benchmark method in the same run. Its actual base/head delta was -5.7% with overlapping confidence intervals (within noise). This change makes that distinction explicit and prevents the within-run ranking from being mistaken for a regression.

Tests

  • python3 .mise/tasks/test_generate-benchmark-summary.py
  • python3 .mise/tasks/test_update-benchmarks.py
  • ruff check and ruff format --check on the changed Python files
  • mise run lint (full lint is blocked only by the pre-existing README Slack link returning HTTP 403; the pre-push scoped lint passed)

Related: #2329.

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger
zeitlinger marked this pull request as ready for review August 19, 2026 09:27
@zeitlinger
zeitlinger merged commit e5fa067 into prometheus:main Aug 19, 2026
18 checks passed
@zeitlinger
zeitlinger deleted the agent/benchmark-report-verdict branch August 19, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants