agent-tracing

Inspects agent execution snapshots and replays recorded LLM calls across models.

74|11|Updated Jul 4, 2024
One-click install
npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill agent-tracing-opensourceagi
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: agent-tracing
Source: https://github.com/OpenSourceAGI/qwksearch-research-agent/tree/main/apps/qwk-in-lobe/.agents/skills/agent-tracing
Command: npx skills add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill agent-tracing-opensourceagi

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Debugging agent behavior requires visibility into what the LLM actually saw and did at each step, but production and local agent runs leave no inspectable record of messages, tool calls, or context assembly. This Skill records every agent execution step as a snapshot and provides a CLI to inspect, diff, and replay those traces. ## Core Features & Use Cases - Trace Inspection: Browse execution snapshots with commands for messages, tool calls, events, context engine payloads, available tools, and user memory at any step. - Remote Trace Pulling: Download production or staging snapshots by operation id via lh trace op or a standalone CLI with a configured object-storage base URL. - Context Window Mapping: Render per-call context composition with prefix-cache reuse analysis using ctx-map, including standalone HTML reports. - Model Replay: Re-issue a frozen recorded LLM payload against other models with an llm-rubric judge to test whether a different model would have completed the same run. - Use Case: An agent called the wrong tool in production. Pull the trace by operation id, inspect the step's available tools versus actual tool calls, then replay that call against another model to determine whether the fault lies in context assembly or model selection. ## Quick Start Ask the agent to inspect the latest agent trace and show the messages and tool calls for a specific step.

Frequently Asked Questions about agent-tracing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I inspect what messages were sent to the LLM in an agent run?

Run agent-tracing inspect with the trace id and the -m flag to see the context engine input, params, and final LLM payload for a step. Use --msg N to view the full content of any individual message.

How do I debug wrong or missing tool calls in an agent?

Use agent-tracing inspect -T to list the tools available at a step and -t to see the actual tool calls and results. Comparing the two shows whether the tool was missing from the payload or the model chose not to call it.

How do I pull a production agent trace by operation id?

Use lh trace op inspect with the operation id, which requires only a LobeHub login. The standalone agent-tracing CLI also works if you configure TRACING_BASE_URL with the bucket's public domain.

Can I test whether a different model would have completed the same agent run?

Yes, the replay command re-issues a recorded LLM payload against other models and returns a PASS or FAIL verdict from an llm-rubric judge. Use --all-steps to replay every call in the operation.

Why does replay not reproduce the exact recorded output?

Sampling parameters like temperature and max_tokens are not recorded in snapshots, so replays use the server's current defaults. Model-to-model comparisons remain valid since every target receives the same request.

Where are agent tracing snapshots stored?

Completed local snapshots live in .agent-tracing/ as JSON files, with in-progress partials under _partial/ and downloaded remote snapshots under _remote/. The CLI must be run from the repository root.