Evaluates Azure Foundry hosted agents via two-phase invoke-and-score with built-in evaluators.
npx skills add https://github.com/aiappsgbb/awesome-gbb --skill foundry-evals
Please help me install this Agent Skill. Skill: foundry-evals Source: https://github.com/aiappsgbb/awesome-gbb/tree/main/skills/foundry-evals Command: npx skills add https://github.com/aiappsgbb/awesome-gbb --skill foundry-evals
This Skill helps you reliably evaluate Azure Foundry hosted agents by separating response invocation from scoring, avoiding common endpoint routing and cold-start pitfalls.
Use the foundry-evals skill to evaluate an already-deployed hosted agent against a set of test scenarios and generate Foundry eval scores.
High-intent search queries and answers about installing and using this skill.
Evaluating Azure Foundry hosted agents requires a two-phase workflow that invokes the agent sequentially with cold-start-safe warmup using retry and backoff logic, then scores responses using Foundry built-in evaluators to ensure reliable post-deploy quality measurement.
Scoring tool-grounding and citation quality for deployed agents requires running Foundry built-in evaluators on enriched JSONL datasets containing tool definitions and tool_calls/tool_outputs, separating response invocation from scoring for accurate quality assessment.
Setting up continuous evaluation gates for Azure Foundry agents involves reusing Foundry eval definitions while updating per-run datasets, configuring built-in evaluators including tool evaluators, and ensuring sequential non-concurrent requests with pacing and retries.
Yes, you can use Foundry built-in evaluators with custom JSONL datasets for agent regression testing by enriching dataset fields with tool_definitions and tool_calls/tool_outputs, then updating per-run datasets while reusing Foundry eval definitions.
Azure Foundry agent evaluation can fail due to RBAC misconfigurations or incorrect endpoint routing, which requires using get_openai_client(agent_name=...) for correct agent endpoint invocation and applying RBAC troubleshooting steps for judge model deployment access.
Yes, running Foundry built-in evaluators on hosted agents requires a judge model deployment, along with proper RBAC configuration and correct agent endpoint invocation via get_openai_client(agent_name=...) to ensure successful scoring operations.
nesihaver-ILBuild and orchestrate AI agents on Azure AI Foundry with Python SDKs.
Veloxia-agencyVersion, evaluate, and promote production prompt assets with deployment gates.
romankovsvBuild async HTTP clients in Python with httpx and aiohttp.
trueraInstrument LLM apps to capture traces for evaluation and debugging.
