skills.rest
langwatch
langwatchOFFICIAL
Explore all skills by this author →
GitHub

evaluate-multimodal

Evaluate multimodal AI agents processing images, audio, and PDFs with LangWatch.

3.5k|362|Updated Sep 9, 2023
Search Tags:#agent testing#LLM-as-judge#PDF parsing#image evaluation#audio testing#langwatch#multimodal evaluation
One-click install
npx skills add https://github.com/langwatch/langwatch --skill evaluate-multimodal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-multimodal
Source: https://github.com/langwatch/langwatch/tree/main/skills/_compiled/native/evaluate-multimodal
Command: npx skills add https://github.com/langwatch/langwatch --skill evaluate-multimodal

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langwatch, @langwatch/scenario.

What problem does it solve?

Testing agents that handle images, audio, or documents is hard because text-only metrics cannot judge non-text outputs. This Skill sets up modality-appropriate evaluations using LangWatch's LLM-as-judge with image inputs and Scenario's multimodal testing patterns.

Core Features & Use Cases

  • Modality Detection: Reads your codebase to identify whether the agent processes images, audio, PDFs, or mixed inputs.
  • Image Evaluation: Uses LangWatch LLM-as-judge evaluators that accept image inputs to assess classification, description, and OCR quality.
  • Audio and PDF Testing: Applies Scenario's audio-to-text patterns and document parsing evaluation via the LangWatch experiment SDK.
  • Use Case: You built a document parsing agent and need to verify extraction accuracy. This Skill guides you to load real PDFs, run the pipeline, and score extracted fields with exact match and LLM judges.

Quick Start

Ask the agent to evaluate your multimodal agent that processes images and PDFs using LangWatch evaluations.

Frequently Asked Questions about evaluate-multimodal

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an AI agent that processes images?▼

Use LangWatch's LLM-as-judge evaluators, which accept image inputs directly. Load test images, run your agent on each, then call experiment.evaluate with the image path as input and a judge prompt assessing output quality.

How to test audio transcription or voice agents?▼

Use Scenario's audio testing patterns fetched via the langwatch scenario-docs multimodal/audio-to-text command. It covers audio-to-text transcription accuracy checks and audio-to-audio voice agent response verification.

Can I evaluate PDF parsing accuracy with LangWatch?▼

Yes. Follow the PDF parsing evaluation pattern: load documents, run your extraction pipeline, and score results with the LangWatch experiment SDK. Use exact match for structured fields and LLM judges for summaries.

What dependencies does multimodal evaluation require?▼

It requires the LangWatch SDK and the langwatch CLI for fetching documentation. The @langwatch/scenario package is optional and needed for Scenario-based audio and file analysis testing patterns.

Why should I not use text-only metrics for multimodal agents?▼

Text-only metrics cannot judge image, audio, or document content, so they miss modality-specific failures. Use image-aware LLM judges and real file formats rather than synthetic text descriptions of the inputs.

Related Skills

Explore More Software Engineering Skills→
opendatahub-ioopendatahub-io
36

eval-run

Execute skills against test cases, score outputs with judges, and generate run summaries.

Official
Advanced
muratcankoylanmuratcankoylan
17.7k

reasoning-trace-optimizer

Analyze interleaved thinking traces to detect failure patterns and optimize prompts.

Community
Advanced
mguinadamguinada

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry instrumentation.

Community
Advanced
KingxiaoKingxiao

agentforge-benchmark

Design and run structured benchmarks and acceptance criteria for AI agents.

Community
Advanced