skills.rest
understudylabs
understudylabsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

ingest-traces

Ingest production LLM traces into redacted eval sets and cost profiles.

10|5|Updated Jun 3, 2026
Search Tags:#llm#profiling#manifests#privacy#traces#ingest#local-first
One-click install
npx skills add https://github.com/understudylabs/understudy-agent-tools --skill ingest-traces
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ingest-traces
Source: https://github.com/understudylabs/understudy-agent-tools/tree/main/skills/ingest-traces
Command: npx skills add https://github.com/understudylabs/understudy-agent-tools --skill ingest-traces

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

## What problem does it solve? Ingest traces from production LLM workloads into local, redacted eval sets and cost profiles, enabling private, reproducible evaluation and cost analysis. ## Core Features & Use Cases

  • Ingest gateway captures, provider logs, or object-store exports into local traces for analysis.
  • Produce deterministic slices, redacted manifests, and cost-aware profiling to compare local vs frontier models.
  • Use cases include turning a fleet of traces into eval sets for benchmarking, safety testing, and cost analysis. ### Quick Start Start by attaching a local harness to your traces and running the ingest workflow to produce a local eval set.

Frequently Asked Questions about ingest-traces

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest production LLM traces to create private evaluation sets?▼

To ingest LLM traces for private evaluation, this skill processes gateway captures, provider logs, and object-store exports into local, redacted eval sets. It produces deterministic slices and redacted manifests for safe benchmarking.

What are redacted manifests and how do they protect privacy during trace ingestion?▼

Redacted manifests protect privacy by avoiding raw request bodies and using relative paths. This approach emits profiles and manifests that can be reviewed before any external processing, ensuring sensitive data stays local.

Can I use object-store exports and provider logs to profile LLM costs locally?▼

Yes, you can ingest provider logs and object-store exports to produce cost-aware profiles. This allows you to compare local versus frontier models by analyzing deterministic slices from your production workloads.

How do I build reproducible eval sets from gateway captures without exposing raw data?▼

You can build reproducible eval sets from gateway captures by enforcing strict privacy through redacted manifests. The ingestion workflow operates locally to generate deterministic slices without retaining raw bodies.

What is the best way to turn production LLM workloads into local traces for safety testing?▼

The best way to turn production workloads into local traces for safety testing is using a local-first ingestion workflow. It transforms captures into deterministic slices and redacted profiles for private evaluation.

Related Skills

Explore More Software Engineering Skills→
zjykzkzjykzk

data-ingest

Ingest raw text data into an Obsidian wiki as searchable knowledge.

Community
Intermediate
understudylabsunderstudylabs
10

run-local-model-lab

Compare local MLX models against frontier gateways on latency, cost, and quality.

Official
Advanced
tadod12tadod12

nemo-evaluator-sdk

Evaluate LLMs across 100+ benchmarks from 18+ harnesses.

Community
Intermediate
anhtesteranhtester
129

QA Automation Engineer

Generate manual test cases, automation scripts, and API tests from requirements.

Community
Advanced