skills.rest
UKGovernmentBEIS
UKGovernmentBEISOFFICIAL
Explore all skills by this author →
GitHub

ci-maintenance-workflow

Diagnose and fix failing CI tests, smoke tests, and slow test markers in GitHub Actions workflows.

657|419|Updated Oct 2, 2024
Search Tags:#Debugging#ci#smoke test#PyTest#GitHub actions#Code review#Pull request
One-click install
npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill ci-maintenance-workflow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ci-maintenance-workflow
Source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/.claude/skills/ci-maintenance-workflow
Command: npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill ci-maintenance-workflow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Maintaining a large evaluation repository means constantly triaging failing CI runs: broken pytest tests, smoke tests that fail on --limit 0, unmarked slow tests that fail the build, and PRs that need standards review. This Skill provides structured, repeatable workflows for each of these maintenance tasks so fixes are diagnosed, verified, and submitted as proper pull requests.

Core Features & Use Cases

  • Fix Failing Tests: Diagnose a failing pytest test from a GitHub Actions URL using gh run view, reproduce it locally with uv run pytest, fix it, and open a PR with the required checklist.
  • Fix Failing Smoke Tests: Reproduce smoke test failures with tools/run_evals.py, apply root-cause fixes under strict rules (no skipping or xfailing tests), and raise a draft PR when blocked.
  • Mark Slow Tests: Parse CI logs with fetch_slow_tests.py to find tests exceeding the 10-second threshold and add @pytest.mark.slow(N) markers.
  • Automated PR Review: Check pull requests against agent-runnable standards in EVALUATION_CHECKLIST.md and write a structured SUMMARY.md review comment.
  • Use Case: A CI run fails because an eval's smoke test broke after an upstream dataset change. Provide the failing run URL, and the workflow reproduces the failure, identifies the root cause, applies a minimal verified fix, and opens a PR documenting the bug, hypothesis, fix, and validation.

Quick Start

Ask the assistant to fix the failing smoke test from this GitHub Actions run URL and open a pull request with the fix.

Frequently Asked Questions about ci-maintenance-workflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix a failing test from a GitHub Actions CI run?▼

Provide the GitHub Actions run URL, then use gh run view to inspect the failure, reproduce it locally with uv run pytest tests/<eval> --runslow, apply a fix, and open a PR. The PR body must include the full checklist from the repository pull request template.

How do I fix a failing smoke test in an eval repository?▼

Reproduce the failure with uv run python tools/run_evals.py --pred <eval_name> --fail-fast, form a hypothesis from the error and recent git history, then apply a minimal root-cause fix and verify with the same command. Skipping or xfailing the test is explicitly forbidden as a fix.

How do I add pytest slow markers to tests exceeding CI time thresholds?▼

Run fetch_slow_tests.py with the GitHub Actions run URL and a 10-second threshold to get durations and recommended marker values. Add @pytest.mark.slow(N) with the rounded-up maximum duration to each unmarked test, then lint and commit.

What happens when a smoke test failure cannot be reproduced locally?▼

The workflow still produces a draft PR rather than abandoning the run. The PR body documents the commands run, why reproduction or verification failed, and hypotheses about environment or dependency discrepancies, so a human can take over.

Can the PR review workflow replace human code review?▼

No, it only checks code-level standards that can be verified by reading the code, such as agent-runnable checklist items and test coverage gaps. Human-required checks, evaluation report quality, and actual execution still need manual review.

Related Skills

Explore More Software Engineering Skills→
TermiX-officialTermiX-official
109

github

Automate GitHub operations via the gh CLI for PRs, issues, CI runs, and API queries.

Official
Advanced
kortix-aikortix-ai
20.2k

changelog-sync

Generates release notes from merged GitHub PRs and opens changelog pull requests.

Official
Advanced
felixminfelixmin

hlrp-code-quality

Run and repair HLRP code-quality checks with ruff, black, and pytest.

Community
Advanced
Pear-CommercePear-Commerce

tech-log-test-failure-watcher

Monitor Slack #tech-log for api.pearcommerce.com test failures and triage them.

Official
Advanced