Score agent outputs against expected results using an LLM judge.
npx skills add https://github.com/mofa-org/mofa-skills --skill mofa-eval
Please help me install this Agent Skill. Skill: mofa-eval Source: https://github.com/mofa-org/mofa-skills/tree/main/_unpublished/mofa-eval Command: npx skills add https://github.com/mofa-org/mofa-skills --skill mofa-eval
This skill automatically scores and tracks the quality of agent outputs against a defined rubric, enabling rapid detection of regressions and consistency issues across runs.
Run a single evaluation by piping a JSON object with run_id, expected, and actual to mofa-eval evaluate_response.
High-intent search queries and answers about installing and using this skill.
LLM-as-a-judge evaluation scores actual agent outputs against expected results using a defined rubric, returning a structured JSON score via the OpenAI API for automated quality assessment.
Regression testing across runs is tracked by persisting every evaluation in SQLite by run_id, enabling compare_runs and score_summary functions to detect performance changes and consistency issues over time.
Interfacing with the OpenAI API is required for the LLM-driven judge to score agent outputs against expected results, meaning valid OpenAI API credentials are necessary for execution.
Batch evaluation of agent outputs is supported via batch_eval, which scores multiple test cases against expected results and persists every evaluation in SQLite by run_id for audit trails.
Rubrics guide the LLM judge by defining scoring criteria, loaded from the styles directory or using a built-in default, to evaluate actual agent outputs against expected results accurately.
Persisting evaluation results by run_id in SQLite creates an audit trail and enables trend analysis, allowing compare_runs to detect regressions and consistency issues in agent output quality over time.
project-axConfigure AX personal AI agent profiles, LLM providers, and API keys.
itsadijmbtRun MLflow GenAI evaluations with dataset-driven scoring and built-in scorers.
yiwei79Evaluate and refine agent outputs through structured reflection loops.
Sir-chawakornBuilds an LLM evaluation harness using golden datasets and code-based scorers.
