llm-pipeline

Extracts topics and atoms from Telegram messages for RAG context.

Updated Aug 27, 2026
One-click install
npx skills add https://github.com/djimontyp/task-tracker --skill llm-pipeline
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: llm-pipeline
Source: https://github.com/djimontyp/task-tracker/tree/main/.claude/skills/llm-pipeline
Command: npx skills add https://github.com/djimontyp/task-tracker --skill llm-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pydantic-ai, openai, ollama, and includes references (resource) components.

What problem does it solve?

This Skill solves the challenge of extracting meaningful, structured knowledge from unstructured message streams like Telegram chats, turning conversational data into actionable insights.

Core Features & Use Cases

  • Intelligent Message Scoring: Automatically evaluates message importance using multiple factors to filter noise.
  • Knowledge Extraction: Uses Pydantic-AI agents to identify topics and atomic knowledge units from relevant content.
  • Semantic Search: Builds RAG-enabled context for intelligent information retrieval.
  • Use Case: Imagine monitoring a busy team chat channel. Use this Skill to automatically identify important discussions, extract key insights, and build a searchable knowledge base from the most valuable conversations.

Quick Start

Use the llm-pipeline skill to analyze the latest 50 messages from our team channel and extract key topics and insights.

Frequently Asked Questions about llm-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract structured knowledge from Telegram messages?

Extract structured knowledge from Telegram messages using Pydantic AI agents to identify topics and atomic knowledge units. The Skill automatically scores message importance, filters noise, and outputs ExtractionResult objects containing topics and atoms for downstream RAG retrieval.

Can I build a searchable knowledge base from unstructured chat data?

Yes. This Skill transforms conversational data into searchable embeddings by extracting key insights, embedding content to 1536-dimension vectors, and integrating with SemanticSearchService. This enables RAG-enabled context construction for intelligent information retrieval across message archives.

How does semantic search work with extracted message content?

Semantic search works by converting extracted topics and atoms into 1536-dimension embeddings, then using SemanticSearchService to match user queries against the embedded knowledge base. This retrieval-augmented generation approach returns contextually relevant results from your Telegram-derived content.

What are the prerequisites for using Pydantic AI with message extraction?

You need Pydantic AI, OpenAI or Ollama for LLM inference, and structured message data from Telegram. The Skill requires these dependencies to initialize agent pipelines, score messages, and generate ExtractionResult outputs with topics and atoms.

Can I use this for batch processing multiple message channels?

Yes. The Skill supports batch embedding and threshold-based knowledge orchestration across multiple Telegram sources. Process sets of messages, extract knowledge from each, embed results, and consolidate into a unified searchable knowledge base.

What's the difference between topic extraction and atomic knowledge units?

Topics are high-level subject categories identified in messages, while atoms are discrete, actionable knowledge units within those topics. Together they provide both conceptual structure and granular facts, enabling flexible querying and context construction for RAG retrieval.