write-conformance-tests

Automate creation and execution of OpenTelemetry GenAI conformance tests.

27|42|Updated May 12, 2026

Search Tags:#test automation #instrumentation #OpenTelemetry #GenAI #conformance testing

Ready to run

npx skills add https://github.com/open-telemetry/opentelemetry-python-genai --skill write-conformance-tests

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: write-conformance-tests
Source: https://github.com/open-telemetry/opentelemetry-python-genai/tree/main/.github/skills/write-conformance-tests
Command: npx skills add https://github.com/open-telemetry/opentelemetry-python-genai --skill write-conformance-tests

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires opentelemetry-instrumentation, pytest, vcr, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the process of writing conformance-test scenarios for OpenTelemetry instrumentation packages, ensuring that the packages meet GenAI semantic conventions.

Core Features & Use Cases

Scenario Authoring: Automatically create tests/conformance/<scenario>.py modules for every semconv operation emitted by the library.
Test Runner: Execute tests/test_conformance.py to run the generated test scenarios and validate against GenAI semantic conventions.
Declared Gaps: Record known gaps as expected_violations in the test scenarios, facilitating their identification and resolution.

Quick Start

Run the write-conformance-tests skill to automatically generate and execute conformance tests for an OpenTelemetry GenAI instrumentation package.

Frequently Asked Questions about write-conformance-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate conformance tests for OpenTelemetry GenAI instrumentation packages?▼

Automate conformance tests for OpenTelemetry GenAI instrumentation packages by generating test scenarios and executing test_conformance.py to validate against GenAI semantic conventions. This ensures both greenfield and ported packages adhere to required instrumentation standards.

What are OpenTelemetry GenAI semantic conventions and why do they matter for instrumentation?▼

OpenTelemetry GenAI semantic conventions are standardized attributes for generative AI operations. Conformance testing validates that instrumentation packages emit these required semconv operations correctly, ensuring consistent observability data across different GenAI libraries.

How do I generate conformance test scenarios for every semconv operation emitted by my library?▼

Generate conformance test scenarios automatically by creating tests/conformance/<scenario>.py modules for every GenAI semconv operation your library emits. This automated scenario authoring ensures comprehensive coverage of all semantic convention requirements.

Does this conformance testing approach work with both greenfield and ported instrumentation packages?▼

Yes, conformance testing applies to both greenfield and ported OpenTelemetry instrumentation packages. It utilizes the opentelemetry-python-contrib package to validate that all package types correctly implement GenAI semantic conventions.

How do I record known gaps when my instrumentation package violates GenAI semantic conventions?▼

Record known gaps as expected_violations directly within the generated test scenarios. This approach facilitates identification and resolution of conformance issues by documenting where the instrumentation package currently deviates from GenAI semantic conventions.

Do I need pytest and vcr to run OpenTelemetry instrumentation conformance tests?▼

Yes, pytest and vcr are required dependencies for executing the conformance test runner. The test suite executes tests/test_conformance.py using pytest, while vcr handles HTTP interaction recording for validating the opentelemetry-instrumentation package.