osdc-observability

Operates and troubleshoots OSDC monitoring and logging pipelines in Kubernetes and Terraform environments.

17|14|Updated Jun 15, 2023
One-click install
npx skills add https://github.com/pytorch/ci-infra --skill osdc-observability
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: osdc-observability
Source: https://github.com/pytorch/ci-infra/tree/main/osdc/.claude/skills/osdc-observability
Command: npx skills add https://github.com/pytorch/ci-infra --skill osdc-observability

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps engineers operate and troubleshoot OSDC observability when monitoring data, system logs, Kubernetes events, credentials, or Grafana Cloud queries are difficult to configure or diagnose.

Core Features & Use Cases

  • Metrics Operations: Configure and troubleshoot Alloy, kube-prometheus-stack, ServiceMonitor and PodMonitor scraping, Mimir remote write, cardinality filtering, alert rules, and GPU metrics.
  • Logging Operations: Manage journal and Kubernetes event pipelines to Grafana Cloud Loki, including structured metadata, label strategy, RBAC isolation, and IPv6 readiness.
  • Querying and Troubleshooting: Retrieve historical logs and metrics, validate credentials and namespaces, diagnose missing data, investigate Alloy resource usage, and resolve deployment issues across OSDC clusters.

Quick Start

Use the osdc-observability skill to diagnose why monitoring metrics or Kubernetes logs are missing from Grafana Cloud for a specified OSDC cluster.

Frequently Asked Questions about osdc-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot missing Kubernetes logs and metrics in Grafana Cloud?

Troubleshooting missing Kubernetes logs and metrics in Grafana Cloud involves validating Grafana Alloy pipelines, verifying kube-prometheus-stack scrape configurations, checking Mimir remote write connectivity, and inspecting RBAC isolation for credential access.

How do I configure Grafana Alloy to scrape GPU metrics on an IPv6-only EKS cluster?

Configuring Grafana Alloy to scrape GPU metrics on an IPv6-only EKS cluster requires setting up ServiceMonitor or PodMonitor resources, tuning relabel configurations, and ensuring the logging pipeline is IPv6-ready. You must also validate cluster-specific deployment behavior for the scraping endpoints.

How does kube-prometheus-stack handle cardinality filtering for Mimir remote write?

kube-prometheus-stack handles cardinality filtering for Mimir remote write by applying relabel configurations to drop unnecessary metrics before they leave the cluster. This prevents excessive metric ingestion and ensures that only required telemetry data is sent to the Grafana Cloud backend.

Can I use Terraform to manage OSDC alert rules and Grafana Cloud credentials?

Yes, you can use Terraform to manage OSDC alert rules and Grafana Cloud credentials. This approach allows you to validate secret-gated deployments, configure structured metadata for Loki logging pipelines, and enforce RBAC isolation across your Kubernetes observability infrastructure.

Why are my Kubernetes event logs missing structured metadata in Grafana Loki?

Kubernetes event logs might be missing structured metadata in Grafana Loki due to misconfigured label strategies or RBAC isolation blocking access in the Alloy pipeline. Validating the journal and Kubernetes event pipeline configurations ensures that structured metadata is correctly attached and queryable.

What are the limitations of using Grafana Alloy for OSDC observability on Kubernetes?

Limitations of using Grafana Alloy for OSDC observability include potential resource exhaustion from high-cardinality scraping, complexities in configuring RBAC isolation for secret-gated deployments, and the need for specific relabel configurations to ensure IPv6-readiness on EKS clusters.