skills.rest
sovr610
sovr610COMMUNITY
Explore all skills by this author →
WebsiteGitHub

Compute/Throughput Baseline & Regression Gate

Detect training throughput and quality regressions with CI-ready baseline comparisons.

Updated Feb 28, 2026
Search Tags:#performance#throughput#baseline#gate#ci#machine-learning
One-click install
npx skills add https://github.com/sovr610/refffiy --skill compute-throughput-baseline-regression-gate
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Compute/Throughput Baseline & Regression Gate
Source: https://github.com/sovr610/refffiy/tree/main/brain-ai-dev/skills/perf-regression-gate
Command: npx skills add https://github.com/sovr610/refffiy --skill compute-throughput-baseline-regression-gate

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, numpy, pyyaml, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a repeatable, machine-readable gate to detect regressions in training throughput and quality, enabling CI to block merges when performance regresses beyond defined tolerances.

Core Features & Use Cases

  • End-to-end gating: environment capture, deterministic micro-benchmarks, MFU estimation, and baseline comparisons keyed by machine_profile.
  • CI workflow generation: templates for CI to fetch baselines, run gates, and update baselines on main.
  • Use cases include measuring tokens_per_sec_p50, step_time_p50, memory, perplexity, and probe accuracy to protect production model training pipelines.

Quick Start

Run the full perf gate locally with python assets/run_template.py --bench --quality --compare to reproduce CI results.

Frequently Asked Questions about Compute/Throughput Baseline & Regression Gate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect ML training throughput regressions in CI?▼

To detect ML training throughput regressions in CI, use a repeatable gate that captures the environment, runs deterministic micro-benchmarks, and compares results against stored baselines keyed by machine profile.

What metrics should a performance baseline gate track for machine learning pipelines?▼

A performance baseline gate for machine learning pipelines should track tokens_per_sec_p50, step_time_p50, memory usage, perplexity, and probe accuracy to protect production model training from regressions.

How do I benchmark PyTorch training performance deterministically?▼

To benchmark PyTorch training performance deterministically, apply CUDA synchronization during micro-benchmarks to ensure repeatable timing measurements and estimate model FLOPs utilization accurately.

Can I use CI workflows to automatically update performance baselines on merge?▼

Yes, you can use CI workflows to automatically update performance baselines by using templates that fetch baselines, run the throughput gates, and update the stored baselines when code merges to main.

Do I need PyTorch and NumPy to run throughput baseline comparisons?▼

Yes, you need PyTorch and NumPy installed to run throughput baseline comparisons, as the gate relies on these dependencies to execute deterministic micro-benchmarks and estimate training performance.

Related Skills

Explore More Software Engineering Skills→
jaechang-hitsjaechang-hits
298

torchdrug

Train GNN models for molecular property prediction using PyTorch and TorchDrug.

Community
Advanced
JamkrisJamkris
86

benchmark

Measure performance baselines and detect regressions across code changes.

Community
Intermediate
K-Dense-AIK-Dense-AI
33.0k

optimize-for-gpu

Transform CPU-bound Python numerical code into CUDA-accelerated versions.

Official
Advanced
aisa-groupaisa-group
90

llm-evaluation

Compute automated metrics and statistical tests for LLM evaluation.

Official
Advanced