skills.rest
terrylica
terrylicaCOMMUNITY
Explore all skills by this author →
GitHub

data-pipeline-monitoring

Monitor Cloud Platform dual-pipeline health, logs, and alerts via scripts.

Updated Nov 4, 2025
Search Tags:#monitoring#alerts#vm#cloud-run#healthchecks#pushover
One-click install
npx skills add https://github.com/terrylica/gapless-network-data --skill data-pipeline-monitoring
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-pipeline-monitoring
Source: https://github.com/terrylica/gapless-network-data/tree/main/.claude/skills/data-pipeline-monitoring
Command: npx skills add https://github.com/terrylica/gapless-network-data --skill data-pipeline-monitoring

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, ulid, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill centralizes monitoring and operational workflows for dual-pipeline data collection systems on Google Cloud Platform, enabling automated health checks, unified log access, and proactive alerting across both batch (Cloud Run) and real-time (VM) components.

Core Features & Use Cases

  • Health checks: verify both batch and real-time pipelines are operational.
  • Log viewing: access Cloud Run and VM systemd logs from a single workflow.
  • Troubleshooting: guided recovery using on-demand references.
  • Progress monitoring: track long-running operations like historical backfills.
  • Service management: restart services, apply fixes, and redeploy as needed.

Quick Start

Run the monitoring workflow to check health across both pipelines, then review logs or trigger alerts as needed.

Frequently Asked Questions about data-pipeline-monitoring

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor dual-pipeline data systems on Google Cloud Platform?▼

Monitor dual-pipeline data systems by running automated health checks across both Cloud Run batch jobs and VM systemd services, accessing unified logs, and triggering alerts through Pushover for operational visibility and fast issue detection.

Can I monitor both Cloud Run and VM pipelines from a single workflow?▼

Yes, this Skill provides unified monitoring for both Cloud Run Jobs and VM systemd services in one workflow, including health checks, log viewing, troubleshooting guides, and cross-channel alerting without switching tools.

How do I set up health checks and alerts for data pipelines on GCP?▼

Configure automated health checks using check_pipeline_health.py and ping_healthchecks.py, manage credentials through Doppler, and send alerts via send_pushover_alert.py to monitor long-running operations and catch failures early.

What's the best way to troubleshoot pipeline failures across batch and real-time systems?▼

Use guided troubleshooting with on-demand references, view logs from both pipelines, track progress on historical backfills, restart services, and apply fixes through unified service management workflows tailored for dual-pipeline architectures.

Do I need specific credentials or tools to monitor GCP pipelines with alerts?▼

You need Doppler-managed credentials for secure secret handling, the requests and ulid Python libraries, and Pushover access for multi-channel alerting; the Skill handles integration across these components.

Can I access VM systemd logs and Cloud Run logs in one place?▼

Yes, the Skill centralizes log access for both VM systemd services and Cloud Run environments through view_logs.sh, eliminating the need to check multiple logging interfaces separately.

Related Skills

Explore More Data & Analytics Skills→
AgentientAgentient
2

Cloud Run Deployment Skill

Automate Cloud Run deployments and traffic management for containerized services.

Official
Advanced
OpenLaborOpenLabor
4

Server Monitoring

Monitor server uptime and performance metrics to detect incidents.

Official
Intermediate
PeterHddPeterHdd
10

engineering-devops-automator

Automate DevOps processes with Terraform, Docker, Kubernetes, and AWS or GCP.

Community
Advanced
sitharaj88sitharaj88
1

setup-monitoring

Configure application monitoring, logging, error tracking, and alerting with Sentry, Datadog, Grafana, Prometheus, and OpenTelemetry.

Community
Advanced