eval-setup

Configure and validate the agent-eval-harness environment with MLflow tracking and API key checks.

36|39|Updated Apr 2, 2026

Search Tags:#workflow #validation #dependencies #config #environment #preflight #mlflow

Ready to run

npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-setup

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-setup
Source: https://github.com/opendatahub-io/agent-eval-harness/tree/main/skills/eval-setup
Command: npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-setup

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires mlflow, pyyaml, anthropic, and includes scripts (resource) components.

What problem does it solve?

Verifies and configures optional components for the agent-eval-harness, including MLflow tracking, API keys, and environment readiness; non-destructive and triggers on common setup signals.

Core Features & Use Cases

Configure MLflow tracking and experiment context
Verify API keys and environment readiness, with non-destructive fixes
Run preflight checks and guide follow-up steps for evaluation pipelines
Support remote MLflow server configuration and per-project pinning

Quick Start

Run /eval-setup to initialize environment checks and configure optional MLflow tracking.

Frequently Asked Questions about eval-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I configure MLflow tracking for my evaluation pipeline?▼

Configure MLflow tracking by running an environment setup script that parses variables, verifies API keys, creates missing directories, and prepares an experiment context non-destructively for local or remote deployments.

What is a preflight check for MLflow and API dependencies?▼

A preflight check validates environment readiness by parsing environment variables, verifying API keys, checking dependency health, and preparing the MLflow experiment context without applying destructive changes.

Can I run environment validation for remote MLflow servers?▼

Yes, environment validation supports remote MLflow server configuration and per-project pinning by parsing environment variables and verifying API keys to ensure readiness across local or remote deployments.

How do I verify API keys and environment readiness for evaluation workflows?▼

Verify API keys and environment readiness by running preflight checks that parse environment variables, confirm dependency health, and prepare the experiment context for your evaluation pipelines.

Does environment setup modify or destroy existing configuration files?▼

No, environment setup is explicitly non-destructive. It safely parses environment variables, checks dependency health, creates missing directories, and prepares the experiment context without overwriting or destroying existing configuration files.