mz-benchmark

Add a Python class extending Scenario in the feature_benchmark scenarios directory and implement a benchmark() method returning a Td fragment with and timing markers. Run it with bin/mzcompose --find feature-benchmark run default --root-scenario=YourScenario.

Use Feature Benchmark for single-operation latency, Scalability Test for TPS under increasing client concurrency, and Parallel Benchmark for sustained latency distributions over time. The choice depends on whether you measure one query, throughput scaling, or degradation over a load period.

The marker starts measurement after it completes, and ends measurement when it completes. For a single query, use a dummy SELECT 1 with so testdrive startup overhead is excluded from the measurement.

Use the guarantees dict for absolute limits like qps and p99 values, or regression_thresholds for relative limits versus a baseline, such as qps at 0.8 of baseline. Scenarios can also be disabled with the @disabled decorator.

OpenLoop sends operations at a fixed rate regardless of latency using Periodic or Gaussian distributions, while ClosedLoop starts the next operation only after the previous one completes. Open-loop reveals queueing under load; closed-loop measures pure sequential latency.