skills.rest
UKGovernmentBEIS
UKGovernmentBEISOFFICIAL
Explore all skills by this author →
GitHub

check-trajectories-workflow

Analyzes agent trajectories from evaluation logs using Inspect Scout scanners.

657|419|Updated Oct 2, 2024
Search Tags:#trajectory analysis#LLM evaluation#scanner#reward hacking#agent trajectories#inspect scout#evaluation logs
One-click install
npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill check-trajectories-workflow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: check-trajectories-workflow
Source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/.claude/skills/check-trajectories-workflow
Command: npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill check-trajectories-workflow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually reviewing agent trajectories from evaluation logs is slow and error-prone. This workflow automates the detection of external failures, formatting issues, reward hacking, and ethical refusals across evaluation samples.

Core Features & Use Cases

  • Default Scanners: Runs five built-in Inspect Scout scanners covering outcome summaries, external failures, formatting failures, reward hacking, and ethical refusals.
  • Custom Scanners: Creates eval-specific scanners wrapped in InspectEvalScanner objects, with configurable invalidation of successes or failures.
  • Result Analysis: Extracts scanner results, analyzes sample validity, and produces a markdown analysis summary per eval and model.
  • Use Case: After running an agentic evaluation, ask the assistant to check the trajectories in a log file; it will scan hundreds of samples, flag reward hacking attempts, and write a summary report.

Quick Start

Ask the assistant to run the Check Agent Trajectories workflow on a specific evaluation log file in the logs directory.

Frequently Asked Questions about check-trajectories-workflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze agent trajectories from an evaluation log file?▼

Run the Check Agent Trajectories workflow by specifying the log file. It executes Inspect Scout scanners via run_all_scanners.py, then extracts results and analyzes sample validity, producing a markdown summary of findings.

What do the default Inspect Scout scanners check for?▼

The default scanners check five conditions: outcome summary, external failure (CAPTCHAs, rate limits, network issues), formatting failure, reward hacking success, and ethical refusal. Custom scanners can be added per evaluation.

How do I add custom scanners for a specific evaluation?▼

Create an eval_scanners.py file under the eval's trajectory_analysis folder, wrapping each scanner in an InspectEvalScanner object added to a SCANNERS list. Run with the -n flag to load them automatically.

How many samples should I scan when checking trajectories?▼

Inspect Evals guidance recommends analyzing at least 100 samples. The workflow performs a dry run first to count samples and asks whether to run all samples or a limited subset via the --limit flag.

What are the limitations of automated trajectory scanning?▼

Automated scanning with Inspect Scout is faster than manual review but may miss nuanced issues that require human judgment. It is best used as a first-pass filter before deeper manual analysis of flagged samples.

Related Skills

Explore More Software Engineering Skills→
langchain-ailangchain-ai
1.1k

LangSmith Evaluators

Create and run LLM-as-Judge and custom code evaluators in LangSmith.

Official
Intermediate
zxdxjtuzxdxjtu
8

eval-fix

Iteratively generate and evaluate fixes until feature gaps converge.

Community
Advanced
SignmanalSignmanal
1

simplify-code

Review and clean up recent code changes with parallel specialized reviewers.

Community
Advanced
anivaranivar

contributor-analyzer

Analyze engineering contributors from commit diffs to generate performance reviews.

Community
Advanced