skills.rest
kortix-ai
kortix-aiOFFICIAL
Explore all skills by this author →
Website𝕏@kortixGitHub

pipeline-health-check

Detects stale tables, row-count anomalies, schema drift, and failed loads in a Postgres warehouse.

20.2k|3.4k|Updated Oct 5, 2024
Search Tags:#data warehouse#anomaly detection#schema drift#Postgres#etl monitoring#gitHub issues#Data Quality
One-click install
npx skills add https://github.com/kortix-ai/suna --skill pipeline-health-check
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pipeline-health-check
Source: https://github.com/kortix-ai/suna/tree/main/packages/starter/templates/marketplace/runtime/skills/pipeline-health-check
Command: npx skills add https://github.com/kortix-ai/suna --skill pipeline-health-check

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Data warehouse issues like stale tables, silent load failures, and schema drift often go unnoticed until they surface as wrong numbers in downstream dashboards. This Skill runs an hourly read-only health sweep across every monitored table in a Postgres schema, catching anomalies early and routing them to the right people.

Core Features & Use Cases

  • Freshness SLA Checks: Compares each table's last-loaded timestamp against its SLA and flags tables whose upstream load jobs have stopped or stalled.
  • Row-Count Baseline Analysis: Trends daily row counts over a trailing 14-day window to detect short loads, skipped batches, or duplicate inserts.
  • Schema Drift Detection: Diffs column names, types, and nullability against the prior known shape to catch upstream source changes.
  • Alerting and Incident Drafting: Posts one alert per anomaly to a chat channel and drafts a GitHub issue with evidence and likely cause for human triage.
  • Use Case: A data team schedules this Skill hourly; when an overnight ETL job silently fails, the sweep flags the stale table, posts the evidence to the alerts channel, and opens a draft incident issue before anyone checks the morning dashboard.

Quick Start

Ask the agent to run a warehouse health check on your schema and report any stale tables, row-count anomalies, schema drift, or failed loads.

Frequently Asked Questions about pipeline-health-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor data warehouse table freshness automatically?▼

Run a scheduled sweep that queries MAX of each table's timestamp column and compares it against a per-table or default freshness SLA. Tables older than their SLA are flagged as freshness anomalies, typically indicating a stopped or silently failing upstream load job.

How to detect schema drift in Postgres tables?▼

Query information_schema.columns for column names, data types, and nullability, then diff the result against the previously recorded shape. Any added, dropped, renamed, or retyped column indicates drift, usually caused by an upstream source schema change.

Can this Skill modify warehouse data or restart failed pipelines?▼

No. The Skill is strictly read-only: every connector call is a SELECT, and it never restarts jobs, re-runs loads, or edits orchestrator configs. It only diagnoses anomalies, posts alerts, and drafts GitHub issues for human triage.

How does row-count anomaly detection work without stored baselines?▼

Baselines are recomputed each run from the warehouse's own history by grouping row counts per day over a trailing 14-day window. Counts far below trend suggest short or skipped loads, while multi-x jumps suggest duplicate or double-inserted batches.

What happens when one table check fails during a sweep?▼

Each table is handled as an independent unit, so a failure or anomaly on one table never blocks checks on the others in the same sweep. Every anomaly gets exactly one alert and one drafted issue per sweep.

Related Skills

Explore More Data & Analytics Skills→
MisLinkMisLink

alerting-irm

Automate Grafana alerting, IRM workflows, and SLO management via YAML or API.

Community
Advanced
myyxfishcatmyyxfishcat
18

openclaw-quickfix

Diagnoses and fixes common OpenClaw Gateway startup and connectivity issues.

Official
Intermediate
redhat-etredhat-et
9

incident-investigator

Orchestrate logs-fetcher, root-cause-analysis, and context-fetcher to investigate incidents.

Official
Intermediate
npownpow
4

monitor

Monitor operational targets and convert health signals into threshold-based status reports.

Community
Advanced