skills.rest
doanchienthangdev
doanchienthangdevCOMMUNITY
Explore all skills by this author →
GitHub

LLMOps

Manages LLM lifecycle including prompts, RAG pipelines, versioning, A/B testing, evaluation, deployment, monitoring, and cost tracking.

4|1|Updated Dec 30, 2025
Search Tags:#prompt engineering#rag#fine-tuning#model evaluation#llmops#llm deployment
One-click install
npx skills add https://github.com/doanchienthangdev/omgkit --skill llmops
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: LLMOps
Source: https://github.com/doanchienthangdev/omgkit/tree/main/plugin/skills/ai-ml/llm-ops
Command: npx skills add https://github.com/doanchienthangdev/omgkit --skill llmops

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langchain, openai, pinecone-client, tiktoken, rank_bm25, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the complexities of deploying, managing, and optimizing Large Language Models (LLMs) in production environments, including prompt engineering, retrieval-augmented generation (RAG), and model evaluation.

Core Features & Use Cases

  • Prompt Management: Register, version, and A/B test prompts for optimal performance.
  • RAG Pipelines: Ingest documents, process them into chunks, and retrieve relevant information for LLM context.
  • LLM Evaluation: Frameworks for assessing model relevance, faithfulness, and other quality metrics.
  • Cost Management: Tools for tracking token usage and estimating operational costs.
  • Use Case: Deploy a customer support chatbot that leverages RAG to answer user queries based on your company's knowledge base, while continuously evaluating and improving prompt performance and cost-efficiency.

Quick Start

Use the LLMOps skill to ingest documents from the 'knowledge_base/' directory into the RAG pipeline.

Frequently Asked Questions about LLMOps

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a RAG pipeline with LangChain and Pinecone for production?▼

To build a RAG pipeline, ingest documents from your knowledge base directory, process them into chunks, and retrieve relevant information using Pinecone and LangChain to provide LLM context for production applications.

What is the best way to manage prompt versioning and A/B testing for LLMs?▼

Prompt management involves registering, versioning, and A/B testing prompts to optimize LLM performance, addressing non-deterministic outputs and context management challenges in production environments.

How do I evaluate LLM relevance and faithfulness after fine-tuning?▼

LLM evaluation frameworks assess model relevance, faithfulness, and other quality metrics to ensure optimal performance during the operational lifecycle of your deployed models.

Can I track token usage and estimate operational costs for my RAG chatbot?▼

Cost management tools track token usage and estimate operational costs for your RAG chatbot, helping you monitor expenses while continuously evaluating and improving prompt performance.

Do I need OpenAI and tiktoken to deploy a customer support chatbot using RAG?▼

Yes, deploying a customer support chatbot with RAG utilizes OpenAI for generation, tiktoken for token counting, and Pinecone for vector storage to answer queries based on your knowledge base.

Related Skills

Explore More Software Engineering Skills→
aisa-groupaisa-group
90

rag-implementation

Build retrieval-augmented generation systems with embeddings, vector stores, and reranking.

Official
Intermediate
Supporter09Supporter09

llama-factory

Configure and troubleshoot LLaMA-Factory LLM fine-tuning workflows end to end.

Community
Advanced
pomazanbohdanpomazanbohdan
16

rag-pipeline

Deploy a Rust-native RAG pipeline with Candle embeddings and hybrid retrieval.

Community
Advanced
hainamchunghainamchung
54

ai-engineer

Design and implement production-grade LLM applications, RAG systems, and intelligent agents.

Community
Advanced