skills.rest
danny0926
danny0926COMMUNITY
Explore all skills by this author →
WebsiteGitHub

pipeline-monitor

Query extraction_log tables to summarize ETL pipeline health and trends.

Updated Feb 27, 2026
Search Tags:#logging#sqlite#etl#data-quality#data-health#pipeline-monitor
One-click install
npx skills add https://github.com/danny0926/NLP-data-for-trading --skill pipeline-monitor-danny0926
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pipeline-monitor
Source: https://github.com/danny0926/NLP-data-for-trading/tree/main/.claude/skills/pipeline-monitor
Command: npx skills add https://github.com/danny0926/NLP-data-for-trading --skill pipeline-monitor-danny0926

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

監控並評估 ETL pipeline 的健康狀態，透過日誌與資料表的分析提供實時與趨勢化的指標，協商快速定位問題。

Core Features & Use Cases

  • 執行歷史與成功率分析：整合 extraction_log 與相關表格，顯示執行次數、成功率與平均信心度，方便監控穩定性。
  • 資料來源新鮮度檢視：追蹤各來源的最近更新時間，快速判斷資料的新鮮度與時效性。
  • 安全與效能建議：提供維護資料品質與 ETL 效能的實用建議，便於日常運維。

Quick Start

在專案資料庫中執行查詢與分析指令，取得 ETL 健康摘要與變更建議。

Frequently Asked Questions about pipeline-monitor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor ETL pipeline health using SQLite logs?▼

Track ETL data freshness by querying extraction logs to identify the most recent update timestamps for each data source. This allows you to evaluate data timeliness and quickly determine whether your pipeline is meeting operational latency requirements.

What is the best way to analyze ETL execution history and success rates?▼

Analyze ETL data quality by reading the database schema and computing status distributions from extraction logs. This evaluates pipeline health across sources and time ranges, yielding recommended actions for maintaining data quality and ETL performance.

How do I track data source freshness in an ETL pipeline?▼

Track ETL data freshness by querying extraction logs to identify the most recent update timestamps for each data source. This allows you to evaluate data timeliness and quickly determine whether your pipeline is meeting operational latency requirements.

Does this ETL monitoring approach require a specific database schema?▼

Yes, ETL monitoring requires a database schema containing an extraction_log table and related tables. The process reads this schema to compute status distributions, track freshness, and generate recommended actions for maintaining data quality and performance.

Why does my ETL pipeline show low data health confidence scores?▼

Low ETL data health confidence scores appear when extraction_log analysis reveals high failure rates or stale data sources. Computing status distributions and comparing source freshness against time ranges identifies the problematic pipelines causing the decline.

Related Skills

Explore More Data & Analytics Skills→
KaiserWhoLearnsKaiserWhoLearns

sqlite-map-parser

Parse SQLite databases into structured JSON with schema and foreign-key resolution.

Community
Intermediate
nirholasnirholas
71

metrics

Collect, store, export, and query system metrics via a TypeScript API.

Community
Advanced
Datus-aiDatus-ai
1.5k

data-profiler

Profile data quality metrics and value distributions for tabular datasets.

Official
Basic
erichowenserichowens
181

data-pipeline-engineer

Automate ETL/ELT pipeline design, validation, and operation across medallion architectures.

Community
Advanced