skills.rest
amd
amdOFFICIAL
Explore all skills by this author →
WebsiteGitHub

testing-the-gaia-agent

Test the GAIA agent end-to-end through the Go TUI control API.

1.6k|168|Updated Dec 16, 2024
Search Tags:#testing#regression#end-to-end#tui#permission gate#gaia agent#capability ladder
One-click install
npx skills add https://github.com/amd/gaia --skill testing-the-gaia-agent-amd
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: testing-the-gaia-agent
Source: https://github.com/amd/gaia/tree/main/.claude/skills/testing-the-gaia-agent
Command: npx skills add https://github.com/amd/gaia --skill testing-the-gaia-agent-amd

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Validating the flagship GAIA agent is error-prone: plausible answers can hide missing tools, concurrent TUI instances corrupt each other's state, and test runs can pollute the user's real memory database. This Skill provides a proven methodology for driving the agent through the TUI and verifying results against ground truth. ## Core Features & Use Cases - Capability Ladder: A seven-rung test sequence (L1-L7) from basic arithmetic through skill loading to real GitHub triage, with pass conditions and reference timings for each rung. - Environment Isolation: Mandatory launcher configuration using GAIA_TUI_HOME, GAIA_MEMORY_DB, and GAIA_AGENT_LOG to prevent cross-session contamination and protect user data. - Permission Gate Verification: Checks that gh commands land in the correct ALLOW/CONFIRM/REFUSE tiers, including rendered-frame inspection of confirmation modals. - Use Case: After merging feature branches into the TUI, run the ladder to catch regressions like broken Enter handling that unit tests miss, then verify skill tool registration against the agent log. ## Quick Start Ask the agent to run the capability ladder against the GAIA TUI and verify each rung's output against ground truth from the gh CLI.

Frequently Asked Questions about testing-the-gaia-agent

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test the GAIA agent through the TUI?▼

Build the TUI with go build, launch it via a PowerShell launcher that sets GAIA_TUI_HOME, GAIA_MEMORY_DB, and GAIA_AGENT_LOG, then drive it with util/tui_driver.py through the control API. Run the seven-rung capability ladder in order and stop at the first failure.

How do I verify the agent is not fabricating tool results?▼

Compare every capability claim against ground truth from outside the agent, such as running gh issue list directly and matching issue numbers exactly. If a result cannot be independently verified, report it as unverified rather than passing.

Why is the GAIA agent suddenly 2-5x slower during testing?▼

A concurrent gaia eval agent run or a second TUI instance competes for the single-slot Lemonade backend, inflating every turn. Check running python processes for eval activity and kill extra gaia-drive.exe instances before blaming the product.

Can I run multiple TUI instances at the same time?▼

No. Two TUIs overwrite each other's control.json discovery file, compete for the model, and cause cross-session contamination. GAIA_TUI_HOME isolates the discovery file but does not remove model contention, so always run exactly one TUI.

Why does the agent have no skills on a clean checkout?▼

The flagship ships with no skills staged; gaia_agent/skills/ contains only .gitkeep and all skill keys in gaia-agent.yaml are commented out. Copy the skill from hub/skills into ~/.gaia/skills/ rather than using gaia skill import, which re-stamps the tier.

How do I test the gh permission gate without an LLM?▼

Run python util/tui_driver.py gate, which prints each command case with its tier and ok/WRONG status instantly with no model involved. Expect 13/13 as expected, then confirm end-to-end in the TUI that CONFIRM commands show a modal and REFUSE commands never prompt.

Related Skills

Explore More Software Engineering Skills→
Lingtai-AILingtai-AI
692

lingtai-update

Routes install, update, detection, and diagnosis tasks for lingtai-tui and lingtai-portal binaries.

Official
Advanced
Patasse97Patasse97

run

Orchestrate the full AgentHub lifecycle in a single command.

Community
Advanced
toqoztoqoz

tui-acceptance-checks

Drives CLI programs through tmux panes and asserts on captured output.

Community
Advanced
asafelobotomyasafelobotomy
7

webapp-testing

Set up Playwright end-to-end tests for web applications.

Community
Intermediate