Authors Vally conformance test stimuli for prompts, instructions, agents, and skills.
npx skills add https://github.com/microsoft/hve-core --skill vally-tests
Please help me install this Agent Skill. Skill: vally-tests Source: https://github.com/microsoft/hve-core/tree/main/.github/skills/hve-core/vally-tests Command: npx skills add https://github.com/microsoft/hve-core --skill vally-tests
Writing conformance tests for AI artifacts (prompts, instructions, agents, skills) is repetitive and error-prone: each test needs the right grader, the right eval file, dedupe against existing stimuli, and a safety check to keep adversarial content out of the eval suite. This Skill standardizes that authoring pipeline so every stimulus follows the same five-step workflow.
.prompt.md, .instructions.md, .agent.md, or SKILL.md file and generate conformance stimuli routed to the correct Vally eval YAML with an appropriate grader (prompt, output-contains, output-matches)..agent.md file; it detects the kind, selects checks from the agents reference, picks graders, runs the safety self-check, and appends advisory stimuli to evals/agent-behavior/stimuli/<slug>.yml.Ask the assistant to author Vally conformance tests for a specific artifact file such as .github/agents/hve-core/rpi-agent.agent.md.
High-intent search queries and answers about installing and using this skill.
Point the skill at the artifact file (.prompt.md, .instructions.md, .agent.md, or SKILL.md). It detects the artifact kind, loads the matching per-kind reference, selects a Vally grader, runs a safety self-check, and appends a dedupe-checked stimulus block to the routed eval YAML file.
Use the corpus-import mode with a file matching the template columns: prompt, kind, target_artifact, grader, tags, expected_refusal_category, notes. The import_corpus.py script validates each row, runs the safety lint, dedupes by SHA-256 hash, and emits an append-only YAML patch plus a JSON report.
Vally CLI 0.9.0 supports type: prompt (LLM-scored rubric), output-contains (literal substring), and output-matches (regex), each with negated forms. The json_schema grader is not shipped in 0.9.0; the skill recommends a regex envelope workaround until it ships.
It refuses seven categories: jailbreak, prompt-injection, harmful-elicitation, TOS violation, Code of Conduct violation, model-refusal-elicitation, and PII or secret extraction. Matching requests receive a canonical refusal block pointing to the Code of Conduct or RAI Planner agent.
Every stimulus is tagged tags.advisory: true, so failures appear in CI summaries and per-trial JSONL output without failing the build. Graduation to authoritative requires at least 30 CI runs, a 5% false-positive ceiling, CODEOWNERS sign-off, and a CHANGELOG entry.
The skill requires Vally CLI 0.9.0 or later, PowerShell 7+, bash, and Python 3.11+ with uv for corpus-import workflows. The Python import path depends on openpyxl for XLSX files, with pytest and ruff used for development.
ivanrvpereiraApply test-driven RED-GREEN-REFACTOR cycles to author and verify Skills.
0xHoneyJarExecute Loa framework test suites and output per-trial JSONL results.
adelproAnalyze adversary tactics with MITRE ATT&CK to improve detection and response.
dotnetScaffolds eval.yaml evaluation specs for agent skills in the dotnet/skills repository.