skills.rest
Arasz
AraszCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

dotnet-ai-evaluation

Evaluates LLM response quality in .NET using Microsoft.Extensions.AI.Evaluation judge models.

2|Updated Aug 2, 2026
Search Tags:#llm-as-judge#llm evaluation#dotnet#response quality#evaluators#lm studio#microsoft.extensions.ai
One-click install
npx skills add https://github.com/Arasz/ai-raccoon --skill dotnet-ai-evaluation-arasz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dotnet-ai-evaluation
Source: https://github.com/Arasz/ai-raccoon/tree/main/.ai-badger/skills/learned/uncategorized/dotnet-ai-evaluation
Command: npx skills add https://github.com/Arasz/ai-raccoon --skill dotnet-ai-evaluation-arasz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Scoring LLM chat completions for quality in .NET requires navigating the Microsoft.Extensions.AI.Evaluation 10.8 API surface, judge model configuration, and undocumented wire behaviors that cause silent failures like null metrics and misrouted local models. ## Core Features & Use Cases - LLM-as-judge evaluation: Score responses with Relevance, Groundedness, Completeness, Coherence, Fluency, and Equivalence evaluators against OpenAI-compatible or local LM Studio judges. - Verified 10.8 API guidance: Correct client construction, EvaluationContext subclassing, temperature pinning via decorating IChatClient, and the tag-based response protocol evaluators actually send. - Local judge operations: LM Studio probing, reasoning-model traps, memory guardrails, silent model-fallback detection, and a judge-quality agreement method with an 80% known-good bar. - Use Case: A developer building a nightly trend-not-gate eval job for a .NET AI assistant uses this to wire a judge client, replay frozen fixtures through evaluators, and validate a local gemma judge before adoption. ## Quick Start Use the dotnet-ai-evaluation skill to set up a Microsoft.Extensions.AI.Evaluation judge that scores my chat completion responses for relevance and groundedness.

Frequently Asked Questions about dotnet-ai-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM response quality in .NET?▼

Use Microsoft.Extensions.AI.Evaluation Quality evaluators (Relevance, Groundedness, Completeness, Coherence, Fluency, Equivalence) via EvaluatorExtensions.EvaluateAsync with a ChatConfiguration wrapping an IChatClient judge. Results arrive as NumericMetric values in EvaluationResult.Metrics.

How to use LM Studio as a local judge for .NET AI evaluation?▼

Point Microsoft.Extensions.AI.OpenAI at http://localhost:1234/v1 with any placeholder key and call GetChatClient(modelId).AsIChatClient(). Verify the response model field matches the request per call, since LM Studio silently serves whatever is loaded when the requested model is missing.

Why does my evaluator return a null metric value?▼

Evaluators cap max_completion_tokens at 800, so small local judges truncate mid-reasoning with finish_reason length and never emit the S2 score tag, leaving NumericMetric.Value null. Raise the cap to about 2000 via a decorating IChatClient and increase the HttpClient timeout.

Does Microsoft.Extensions.AI.Evaluation send json_schema response_format?▼

No. The 10.8 Quality evaluators send response_format type text and parse scores from S0/S1/S2 tags, while Equivalence expects a bare integer 1-5. A fake judge in tests must mimic that protocol rather than returning JSON.

Can a reasoning model like bonsai-27b work as a judge?▼

Yes, but only after applying the LM Studio UI Reasoning Disabled preset, since per-request reasoning off is ignored on the OpenAI-compatible endpoint. Without it, output lands in reasoning_content and content is empty.

Should judge calls run in the live request path?▼

No. Inline judging doubles latency and LLM cost with no actionable consumer at request time. Judges belong offline in nightly fixed-scenario CI runs and dev-time on-demand evaluations.

Related Skills

Explore More Software Engineering Skills→
jokerman89jokerman89

eval

Runs a voice test rubric against a voice corpus and writes per-cell calibration results.

Community
Intermediate
LauraFlorentinLauraFlorentin
1

evaluation

Monitors agent performance and generates metrics for CI/CD pipelines and audits.

Community
Intermediate
ACGSpgpACGSpgp

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Official
Advanced
ContextJet-aiContextJet-ai
29

add-llm-evals

Add offline and online evaluation scoring to LLM and agent applications.

Official
Advanced