bench-compare

Compare Pinot JMH benchmark results between baseline and current codebases.

6.1k|1.5k|Updated May 19, 2014

Search Tags:#performance #benchmarking #java #git-worktree #jmh #pinot #bench-compare

Ready to run

npx skills add https://github.com/apache/pinot --skill bench-compare-apache

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: bench-compare
Source: https://github.com/apache/pinot/tree/main/.claude/skills/bench-compare
Command: npx skills add https://github.com/apache/pinot --skill bench-compare-apache

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps you compare Pinot JMH benchmark results between a baseline commit and the current codebase, surfacing performance deltas in a single, easy-to-interpret report.

Core Features & Use Cases

Baseline vs Current: Automatically builds and runs benchmarks in isolated git worktrees for two perspectives and outputs comparative results.
Risk Detection: Flags substantial changes where the delta exceeds the measurement error to highlight meaningful performance shifts.
Reproducible Workflow: Produces two result files and a diff, enabling easy sharing in PRs and reproducibility across environments.

Quick Start

Run a controlled side-by-side Pinot performance comparison between baseline and current code by generating both results and a diff table.

Frequently Asked Questions about bench-compare

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I compare JMH benchmark results between two commits?▼

To compare JMH benchmark results, this skill builds and runs benchmarks in isolated git worktrees for a baseline commit and your current codebase, then generates a side-by-side diff report of the performance metrics.

What is the best way to validate Pinot performance changes in a PR?▼

Validating Pinot performance changes requires running JMH benchmarks against a baseline and current version, outputting two result files and a diff table to highlight substantial shifts exceeding measurement error for easy PR sharing.

Do I need a Maven build to run Pinot benchmark comparisons?▼

Yes, you need a Maven-based Pinot build. The workflow requires proper JMH invocation via org.openjdk.jmh.Main and safeguards to avoid stale dependencies, ensuring deterministic builds across both baseline and current runs.

How does benchmark risk detection work for performance deltas?▼

Benchmark risk detection flags substantial changes where the performance delta exceeds the measurement error, highlighting meaningful performance shifts between the baseline commit and the current codebase in the final comparative report.

Can I use git worktrees to isolate baseline and current benchmark runs?▼

Yes, the workflow applies isolated git worktrees to separate the baseline commit and current codebase perspectives, ensuring deterministic builds and preventing cross-contamination of dependencies during JMH benchmark execution.

Why are my benchmark results inconsistent across different code versions?▼

Inconsistent benchmark results often stem from stale dependencies or non-deterministic builds. This skill applies safeguards against stale dependencies and uses isolated worktrees to ensure reproducible JMH benchmark comparisons.