skills.rest
Notysoty
NotysotyCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

LLM Tracing and Observability Setup

Configure end-to-end tracing for LLM applications with Python and TypeScript.

9|1|Updated Mar 15, 2026
Search Tags:#llm#observability#trace#opentelemetry#langfuse#langsmith#helicone
One-click install
npx skills add https://github.com/Notysoty/openagentskills --skill llm-tracing-and-observability-setup
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: LLM Tracing and Observability Setup
Source: https://github.com/Notysoty/openagentskills/tree/main/skills/llm-tracing-setup
Command: npx skills add https://github.com/Notysoty/openagentskills --skill llm-tracing-and-observability-setup

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

End-to-end observability for LLM pipelines enables you to see prompts, model responses, tool calls, latency, and costs, eliminating blind debugging and guesswork.

Core Features & Use Cases

  • Backend-agnostic tracing setup supporting LangSmith, Langfuse, Helicone, and OpenTelemetry.
  • Automatic, structured tracing of LLM calls, prompts, model outputs, and token counts, plus tool calls and retrieval steps.
  • Easy integration with LangChain and other LLM stacks in Python or TypeScript, with guidance for instrumentation and metadata tagging.
  • Production-readiness guidance including metrics to monitor and steps for adding custom spans and user feedback.

Quick Start

Copy this file to .agents/skills/llm-tracing-setup/SKILL.md in your project root and follow the prompts to enable end-to-end LLM observability with your chosen tracing backend.

Frequently Asked Questions about LLM Tracing and Observability Setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up end-to-end tracing for my LLM application?▼

End-to-end tracing for LLM applications is configured by selecting a backend like Langfuse or LangSmith and instrumenting prompts, responses, tool calls, and metadata to capture latency, token counts, and costs across environments.

Can I use OpenTelemetry to monitor LLM pipeline costs and latency?▼

Yes, OpenTelemetry is supported as a backend to monitor LLM pipelines, enabling you to collect latency, token counts, and cost data by adding custom spans and tagging traces with environment and pipeline version.

Does this LLM observability setup work with Python and TypeScript stacks?▼

This LLM observability setup supports both Python and TypeScript, offering easy integration with LangChain and other LLM stacks to provide structured tracing of model outputs and tool calls.

What is the best way to add custom spans to LLM traces?▼

The best way to add custom spans to LLM traces is by following the production-readiness guidance provided by your chosen backend, which includes steps for metadata tagging and capturing specific retrieval steps.

Which tracing backends are supported for LLM observability?▼

Supported tracing backends for LLM observability include LangSmith, Langfuse, Helicone, and OpenTelemetry, allowing backend-agnostic setup for automatic structured tracing of model calls and outputs.

Why do I need to tag traces with environment and pipeline version?▼

Tagging traces with environment and pipeline version satisfies production-readiness requirements, allowing you to filter observability data, monitor specific metrics, and debug issues across multiple deployment contexts.

Related Skills

Explore More Software Engineering Skills→
slysikslysik

instrumenting-with-mlflow-tracing

Instrument Python and TypeScript code with MLflow Tracing.

Community
Intermediate
qcmuuqcmuu

langsmith-observability

Trace LLM application runs and evaluate model behavior with LangSmith.

Community
Intermediate
Clay-HHKClay-HHK
2

langsmith-observability

Trace LLM calls and collect inputs, outputs, latency, and metadata.

Community
Advanced
PostHogPostHog
58

omnibus-instrument-llm-analytics

Instrument LLM calls with PostHog LLM Analytics to capture token counts, latency, and costs.

Official
Advanced