llmops

Manage LLM operations with prompt versioning, evaluation pipelines, cost controls, guardrails, and RAG monitoring.

Updated Mar 20, 2026

Search Tags:#llm #evaluation #prompt management #cost optimization #large language models

Ready to run

npx skills add https://github.com/Cure-Consulting-Group/iep-and-thrive --skill llmops-cure-consulting-group

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llmops
Source: https://github.com/Cure-Consulting-Group/iep-and-thrive/tree/main/.agents/skills/llmops
Command: npx skills add https://github.com/Cure-Consulting-Group/iep-and-thrive --skill llmops-cure-consulting-group

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenges of managing Large Language Models (LLMs) at scale, focusing on cost optimization, ensuring high-quality outputs, and maintaining safety and security.

Core Features & Use Cases

Prompt Management: Version control, A/B testing, and regression testing for prompts.
Evaluation Pipelines: Offline and online evaluation with human and automated judgment.
Cost Optimization: Model tiering, caching, and token budget enforcement.
Guardrails and Safety: Input and output validation, rate limiting, and fallback strategies.
RAG Monitoring: Quality metrics, index freshness, and embedding drift detection.
Incident Response: AI-specific incident types and a runbook for effective management.
Use Case: For a company integrating LLMs into their product, this Skill helps ensure efficient operations, high user satisfaction, and regulatory compliance.

Quick Start

To start using the llmops skill, execute the command 'llmops [ai-feature-or-pipeline]'.

Frequently Asked Questions about llmops

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I manage prompt versioning and run regression testing for large language models?▼

Prompt management for large language models involves version control, A/B testing, and regression testing to track changes. This ensures consistent quality when iterating on prompts across different model versions.

What is the best way to optimize LLM costs through model tiering and token budget enforcement?▼

Cost optimization for LLMs uses model tiering, caching, and token budget enforcement to reduce spend. This controls operational expenses by routing requests to appropriate models and limiting token usage.

How do I set up offline and online evaluation pipelines for LLM outputs?▼

Evaluation pipelines for LLM outputs combine offline and online evaluation with human and automated judgment. This validates model quality systematically before and after deployment to catch regressions.

Can I implement input validation and rate limiting guardrails for AI features?▼

Guardrails and safety for AI features include input and output validation, rate limiting, and fallback strategies. This prevents malicious prompts, blocks unsafe content, and maintains service availability.

How does RAG monitoring handle embedding drift and index freshness?▼

RAG monitoring tracks index freshness, quality metrics, and embedding drift detection. This ensures retrieval-augmented generation pipelines return accurate, up-to-date information by identifying shifts in vector representations.

Do I need a specific incident response runbook for AI features?▼

AI incident response requires specific runbooks for AI-specific incident types. This manages operational failures effectively by providing structured procedures for handling model outages, safety breaches, and quality drops.