langsmith-observability

Traces, evaluates, and monitors LLM applications for performance and errors.

1|Updated Feb 10, 2026

Search Tags:#debugging #monitoring #llm #evaluation #observability #tracing

Ready to run

npx skills add https://github.com/informatico-madrid/Architect-Expert-Gap-Forge --skill langsmith-observability-informatico-madrid

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: langsmith-observability
Source: https://github.com/informatico-madrid/Architect-Expert-Gap-Forge/tree/main/.github/skills/langsmith
Command: npx skills add https://github.com/informatico-madrid/Architect-Expert-Gap-Forge --skill langsmith-observability-informatico-madrid

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langsmith, and includes references (resource) components.

What problem does it solve?

This Skill provides comprehensive tools for understanding, debugging, and improving Large Language Model (LLM) applications by offering detailed tracing, evaluation, and monitoring capabilities.

Core Features & Use Cases

Tracing: Capture and visualize the execution flow of LLM calls, chains, and agents.
Evaluation: Systematically test and score model performance against datasets using built-in or custom metrics.
Monitoring: Track production LLM systems for performance, errors, and costs.
Use Case: Debug a complex agent by tracing its decision-making process, identify a specific prompt that leads to poor performance, and then create an evaluation dataset to systematically test improvements.

Quick Start

Use the langsmith-observability skill to trace your OpenAI API calls by setting the LANGSMITH_TRACING environment variable to true.

Frequently Asked Questions about langsmith-observability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I trace LLM calls and agent decisions to debug poor performance?▼

Trace LLM calls by setting the LANGSMITH_TRACING environment variable to true. This captures and visualizes the execution flow of prompts, chains, and agents to pinpoint specific decisions causing poor performance.

What is LLM observability and when do I need it for my application?▼

LLM observability is the systematic tracing, evaluation, and monitoring of AI applications. You need it to understand execution flows, systematically test performance against datasets, and track production latency, cost, and errors.

Can I monitor production LLM systems for cost and latency using LangSmith?▼

Yes, you can monitor production LLM systems to track performance metrics, errors, and costs. LangSmith provides built-in capabilities to oversee latency and operational health across major LLM providers.

Does LangSmith observability work with LangChain and major LLM providers?▼

Yes, LangSmith observability integrates with major LLM providers and frameworks like LangChain. This compatibility allows you to trace chains and agents directly within your existing application architecture.

How do I systematically evaluate model performance against custom datasets?▼

Systematically evaluate model performance by creating an evaluation dataset and applying built-in or custom metrics. This allows you to score model outputs, test improvements, and identify prompts causing poor results.