skills.rest
pollinations
pollinationsOFFICIAL
Explore all skills by this author →
Website𝕏@pollinations_aiGitHub

monitor-services

Health-checks and auto-restarts Pollinations GPU inference services across Vast.ai, RunPod, Lambda, and OVH.

5.0k|1.0k|Updated Apr 15, 2021
Search Tags:#Infrastructure#auto-restart#Runpod#GPU monitoring#Vast.ai#Health check#devOps
One-click install
npx skills add https://github.com/pollinations/pollinations --skill monitor-services
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: monitor-services
Source: https://github.com/pollinations/pollinations/tree/main/.claude/skills/monitor-services
Command: npx skills add https://github.com/pollinations/pollinations --skill monitor-services

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Pollinations runs image, video, and music generation on a fleet of distributed GPU backends (Flux on Vast.ai, Z-Image and Klein on RunPod, LTX-2 and ACE-Step on a Lambda GH200, legacy image on OVH). When a pod dies, a GPU falls off the bus, or a tunnel degrades, production requests fail with 524 timeouts. This Skill automates detection and recovery of those failures.

Core Features & Use Cases

  • Fleet-wide health checks: Probes the gen.pollinations.ai worker registry plus per-service health endpoints for Flux, Z-Image, LTX-2, ACE-Step, Klein, DreamShaper, and the legacy OVH image service.
  • Auto-restart and recovery: Restarts systemd units, relaunches pod processes, and walks through full RunPod pod recreation including provisioning, model downloads, and secret pushes.
  • Failure diagnosis: Cross-checks RunPod pod status against the live registry to catch pods that are running and billing but serving zero traffic, and distinguishes tunnel degradation from GPU failure.
  • Use Case: Run it on a recurring loop (e.g. every 30 minutes) so an on-call operator gets a status table of every GPU backend with automatic restart attempts for anything unhealthy.

Quick Start

Ask the AI to run the monitor-services check on all Pollinations GPU backends and report a status table with latency for each service.

Frequently Asked Questions about monitor-services

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I health-check GPU inference servers across multiple cloud providers?▼

Query the central worker registry first, then curl each backend's /health endpoint directly. This Skill checks the gen.pollinations.ai registry plus per-service endpoints on Vast.ai, RunPod, Lambda, and OVH hosts, reporting status and latency per service.

How do I restart a crashed RunPod pod serving an image model?▼

SSH into the pod and run its relaunch script for a dead process, or stop and start the pod with runpodctl if the GPU fell off the bus. Stop/start wipes the container disk, so back up the provision script first and re-provision afterward.

Why does a RunPod pod show RUNNING but serve no traffic?▼

The pod's server process may have died or its GPU may have fallen off the bus while the control plane still reports RUNNING. Cross-check the pod list against the live worker registry; any RUNNING pod absent from the registry is idle and needs a relaunch or restart.

Can health checks pass while the service is still broken?▼

Yes. A worker can heartbeat to the registry while its tunnel is too slow to serve requests, and a heartbeat thread keeps firing even when generation hangs. Always time a real probe of the registered URL and verify old processes actually died after a relaunch.

What credentials are needed to run these service checks?▼

SSH keys are stored in SOPS-encrypted secrets (extracted with sops and jq), the OVH host uses a local key, and end-to-end tests use a test API token from the .testingtokens file. RunPod access uses the account API key from the runpodctl config.

Related Skills

Explore More Software Engineering Skills→
RachasumanthRachasumanth

gpu-provisioner

Provisions cost-efficient GPU instances for LLM training across multiple cloud providers.

Community
Advanced
ilove323ilove323

incident-response

Automate incident response workflows from detection to postmortem analysis.

Community
Advanced
patricio0312revpatricio0312rev
5

db-performance-watchlist

Detect and analyze database performance issues by collecting real-time metrics and highlighting slow queries.

Community
Advanced
drewid74drewid74
8

grafana-prometheus-monitoring

Provision Grafana dashboards, datasources, and Prometheus SLO alerts as code.

Community
Advanced