skills.rest
Arize-ai
Arize-aiOFFICIAL
Explore all skills by this author →
Website𝕏@arizeaiGitHub

genai-conformance

Runs and iterates the OpenInference GenAI conformance harness against OTel semantic conventions.

1.2k|302|Updated Dec 26, 2023
Search Tags:#conformance#openTelemetry#genai#semantic conventions#openinference#LLM observability#Weaver
One-click install
npx skills add https://github.com/Arize-ai/openinference --skill genai-conformance
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: genai-conformance
Source: https://github.com/Arize-ai/openinference/tree/main/.claude/skills/genai-conformance
Command: npx skills add https://github.com/Arize-ai/openinference --skill genai-conformance

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Validating that OpenInference instrumentors correctly dual-write native attributes into OTel GenAI semantic conventions (gen_ai.*) requires running a complex harness with mock providers, Weaver live-check, and careful interpretation of coverage gaps. This Skill guides running, interpreting, and extending that conformance workflow.

Core Features & Use Cases

  • Harness Execution: Runs the conformance harness at python/openinference-instrumentation/scripts/conformance/ via uv run, exercising Anthropic, OpenAI, and Google GenAI instrumentors against deterministic mock APIs.
  • Gap Diagnosis: Categorizes missing gen_ai.* registry attributes into dual-write conversion gaps, test scenario gaps, mock data gaps, unsupported provider features, and application-level attributes.
  • Coverage Iteration: Guides extending _genai_conversion.py, adding unit tests in test_genai.py, and adding providers or scenarios to maximize registry coverage.
  • Version Bumping: Walks the checklist for upgrading pinned Weaver and semantic-conventions versions, including refreshing vendored JSON schemas.
  • Use Case: When a user asks why gen_ai.usage.cache_read.input_tokens is missing from conformance output, the Skill identifies whether the fix belongs in the conversion layer, the mock server, or the test scenario.

Quick Start

Run the GenAI conformance harness and tell me which gen_ai registry attributes are missing and how to fix them.

Frequently Asked Questions about genai-conformance

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run the OpenInference GenAI conformance harness?▼

Run `uv run python/openinference-instrumentation/scripts/conformance/run.py` from the repository root. The first run downloads pinned Weaver and semantic-conventions releases to a local cache; subsequent runs reuse cached artifacts and provider environments.

How do I fix missing gen_ai attributes in conformance output?▼

First categorize the gap: dual-write conversion gaps are fixed in `_genai_conversion.py`, test scenario gaps in the provider conformance script, and mock data gaps in `mock_server.py`. Always add a unit test in `test_genai.py` for conversion changes and re-run the harness.

Which LLM providers does the conformance harness support?▼

The harness supports Anthropic, OpenAI, and Google GenAI through dedicated PEP 723 provider scripts. A single Flask mock server handles all provider endpoints, and new providers are added by appending to PROVIDER_SCRIPTS in run.py.

Why does Weaver report violations on OpenInference native attributes?▼

Weaver flags OpenInference native attributes as missing_attribute violations because they are not part of the OTel registry, which is expected behavior. Genuine shape errors on gen_ai attributes appear as advice with an id other than missing_attribute in live_check.json.

How do I bump the OTel semantic conventions version in the harness?▼

Update SEMCONV_VERSION and WEAVER_VERSION constants in run.py, run the harness to download the new registry, refresh the vendored JSON schemas in tests/fixtures/genai_schemas, run the unit tests, and re-run the harness end-to-end to check for regressions.

Related Skills

Explore More Software Engineering Skills→
Arize-aiArize-ai
1.1k

python-code-reviewer

Review Python OpenInference instrumentation packages for correctness and project standards.

Official
Advanced
KappTech88KappTech88

phoenix-observability

Trace, evaluate, and monitor LLM applications with OpenTelemetry.

Community
Advanced
sgcarstrendssgcarstrends
21

changelog

Generate CHANGELOG.md from conventional commits using semantic-release.

Official
Advanced
golemcloudgolemcloud
1.5k

golem-enable-otlp-scala

Configure the golem-otlp-exporter plugin to export traces, logs, and metrics from Scala Golem agents.

Official
Intermediate