run-eval-harbor

Run and diagnose Harbor evaluations against local or cloud LobeHub targets.

Updated Jan 27, 2026
One-click install
npx skills add https://github.com/SmallAi-API/smaihub --skill run-eval-harbor-smallai-api
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: run-eval-harbor
Source: https://github.com/SmallAi-API/smaihub/tree/main/.agents/skills/run-eval-harbor
Command: npx skills add https://github.com/SmallAi-API/smaihub --skill run-eval-harbor-smallai-api

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires harbor==0.23.0, jinja2, @lobehub/cli, and includes scripts (resource) and references (resource) components.

What problem does it solve? Running Harbor evaluations against a LobeHub deployment requires coordinating infrastructure, credentials, CLI injection, preflight checks, and failure triage across local and cloud targets, which is error-prone when done manually. ## Core Features & Use Cases - Dual-target execution: Run existing Harbor evaluations against either an isolated local production LobeHub stack on port 3210 or a remote LobeHub Cloud target. - Gated preflight and smoke testing: Validate Compose services, gateways, JWKS keys, and endpoints, then run a model-backed hello-world smoke job before any real Harbor run or resume. - Flexible CLI injection: Inject a checkout build from apps/cli or install the published @lobehub/cli npm package inside Harbor task containers via the LhInstalledAgent. - Use Case: An engineer wants to resume a failed Harbor job against their local checkout; the Skill bootstraps the eval infrastructure, verifies preflight and smoke pass, then hands off to the eval repository's own resume command. ## Quick Start Ask the assistant to run an existing Harbor evaluation against the local LobeHub production build using the checkout CLI, providing the agent ID and eval repository path.

Frequently Asked Questions about run-eval-harbor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Harbor evaluation against a local LobeHub build?

Bootstrap the eval infrastructure with scripts/bootstrap.sh, build LobeHub, start the production server on port 3210 with server.sh, then run preflight.sh local and run-smoke.sh local before the eval repository's own job command.

How do I run Harbor evals against LobeHub Cloud?

Provide LH_AGENT_ID and LOBEHUB_CLI_API_KEY in the eval repository's .env file, leave server URLs unset for official Cloud defaults, then run preflight.sh cloud and run-smoke.sh cloud before the real job.

What is the difference between checkout and npm CLI modes?

Checkout mode builds apps/cli locally with pnpm and injects it into Harbor containers via LH_CLI_SOURCE host-dir. Npm mode installs the published @lobehub/cli package inside each Harbor task instead.

Why does Harbor preflight fail on port 3210?

Preflight fails when the production server is not running from this checkout, lacks NODE_ENV=production, or is missing JWKS_KEY. The Skill targets the production build on port 3210, never the dev server on port 3010.

Can this Skill author new Harbor tasks?

No, it only runs and diagnoses existing Harbor evaluations. Task authoring and grading belong to the create-task skill, and product acceptance belongs to the acceptance skill.