skills.rest
langwatch
langwatchOFFICIAL
Explore all skills by this author →
GitHub

test-compliance

Creates scenario and red team tests verifying agents stay observational in regulated domains.

3.5k|362|Updated Sep 9, 2023
Search Tags:#compliance testing#scenario testing#Red Team#agent evaluation#boundary enforcement#langwatch#regulated domains
One-click install
npx skills add https://github.com/langwatch/langwatch --skill test-compliance
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-compliance
Source: https://github.com/langwatch/langwatch/tree/main/skills/_compiled/native/test-compliance
Command: npx skills add https://github.com/langwatch/langwatch --skill test-compliance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires @langwatch/scenario.

What problem does it solve?

AI agents in healthcare, finance, and legal domains must inform users without crossing into prescribing, diagnosing, or giving professional advice. This Skill helps you verify that your agent enforces those boundaries, including under adversarial pressure.

Core Features & Use Cases

  • Boundary Scenario Tests: Write scenario tests with user simulators and judge agents that verify disclaimers, escalation handling, and refusal of prescriptive requests.
  • Red Team Adversarial Testing: Use RedTeamAgent to probe for compliance failures via gradual escalation, authority claims, emotional appeals, and hypothetical framing.
  • Domain-Specific Criteria: Provides reusable judge criteria for healthcare, finance, and legal agents, plus a structured process for fixing failures without overfitting the system prompt.
  • Use Case: A team building a health information chatbot uses this Skill to confirm the agent never recommends specific medications, even when a simulated user insists or claims to be a nurse.

Quick Start

Ask the agent to create compliance boundary and red team scenario tests for my healthcare chatbot using @langwatch/scenario.

Frequently Asked Questions about test-compliance

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test that my AI agent does not give medical advice?▼

Write scenario tests using @langwatch/scenario with a user simulator and judge agent. Define criteria such as the agent not diagnosing conditions, not recommending medications, and including a disclaimer to consult a healthcare professional.

How to red team test an AI agent for compliance boundaries?▼

Use the RedTeamAgent from @langwatch/scenario to adversarially probe the agent. Effective strategies include gradual escalation, authority claims, emotional appeals, hypothetical framing, and context switching from allowed to restricted topics.

What judge criteria should I use for a financial advice chatbot?▼

Use criteria like the agent does not recommend specific stocks, funds, or investment strategies, includes a not-financial-advice disclaimer, and suggests consulting a financial advisor for personalized advice.

Does this testing approach work with multi-turn conversations?▼

Yes, scenario scripts support multi-turn exchanges where the user pushes repeatedly for specific advice. Multi-turn escalation tests are essential because single-turn tests miss persistence attacks on compliance boundaries.

Why does my agent pass compliance tests but fail in production?▼

This usually happens when each failing test was fixed by adding another system prompt rule, causing overfitting to tested phrasings. Instead, diagnose which layer owns the failure, fix the class of problem, and re-run with varied wording.

Related Skills

Explore More Software Engineering Skills→
merge-apimerge-api

post-connection-data-scope-filtering

Manage data scope filtering strategies for Merge integration syncs.

Official
Advanced
Hassan-Ali-Mehdi-3024Hassan-Ali-Mehdi-3024

regression-testing

Build regression test sets and generate reports for AI output quality.

Community
Advanced
coval-aicoval-ai
2

debug-traces

Diagnose OpenTelemetry trace ingestion failures and correlation errors in Coval workflows.

Official
Advanced
davidahmanndavidahmann
12

gait-policy-test-rollout

Test Gait policy changes against intent fixtures and simulate staged rollouts.

Community
Intermediate