monitoring-observability

Name: monitoring-observability
Availability: InStock
Author: Yonatan

Implement Prometheus metrics, Grafana dashboards, and Langfuse tracing for application monitoring.

217|20|Updated Dec 31, 2025

Search Tags:#monitoring #drift detection #llm #observability #grafana #prometheus #langfuse

Ready to run

npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability-yonatangross

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: monitoring-observability
Source: https://github.com/yonatangross/orchestkit/tree/main/plugins/ork/skills/monitoring-observability
Command: npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability-yonatangross

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires prometheus_client, langfuse, scipy, sklearn, numpy, evidently, and includes scripts (resource) and references (resource) and rules (resource) components.

What problem does it solve?

This Skill addresses the challenge of understanding system health and LLM application performance by providing comprehensive patterns for monitoring, observability, and drift detection.

Core Features & Use Cases

Infrastructure Monitoring: Implement Prometheus metrics, Grafana dashboards, and alerting rules for application health.
LLM Observability: Integrate Langfuse tracing for LLM applications, including cost tracking, evaluation scoring, and drift monitoring.
Drift Detection: Apply statistical methods to detect shifts in data distributions and LLM output quality.
Use Case: A developer needs to ensure their FastAPI application is performant and their LLM agent is not hallucinating. They can use this Skill to set up Prometheus metrics for latency and errors, Langfuse traces for LLM calls, and drift detection for input/output data.

Quick Start

Implement Prometheus metrics for my FastAPI application using the RED method.

Frequently Asked Questions about monitoring-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor LLM application performance and detect drift?▼

Detect data drift in LLM outputs using statistical tests like Population Stability Index (PSI) and Kolmogorov-Smirnov (KS) tests, which identify significant shifts in input data distributions and output quality based on historical baselines.

How do I set up Prometheus metrics for a FastAPI application?▼

Implement Prometheus metrics for FastAPI using the RED method to track Rate, Errors, and Duration, instrumenting latency and error rates while managing metric cardinality to prevent database overload.

Can I use Langfuse for LLM observability and cost tracking?▼

Yes, you can integrate Langfuse for LLM observability to implement tracing for LLM calls, track token costs, set up evaluation scoring, and monitor drift in your application's LLM outputs.

What statistical methods are used for drift detection in machine learning?▼

Statistical methods for drift detection in machine learning include Population Stability Index (PSI) and Kolmogorov-Smirnov (KS) tests, which analyze data distributions to detect shifts in model inputs and outputs.

How do I create Grafana dashboards and alerting rules for infrastructure monitoring?▼

Create Grafana dashboards and alerting rules by configuring visualization panels for Prometheus metrics and setting threshold-based alerting strategies aligned with Service Level Objectives (SLOs) for application health.

What are the best practices for managing Prometheus metric cardinality?▼

Best practices for managing Prometheus metric cardinality involve limiting high-dimensional label combinations and avoiding user IDs or request IDs as labels to prevent database bloat and maintain query performance.