sourcea-brain-chatbot-audit

Audits the SourceA Brain public chatbot knowledge pipeline, eval buckets, and gap reports.

Updated Jun 20, 2026
One-click install
npx skills add https://github.com/Noetfield-Systems/SourceA --skill sourcea-brain-chatbot-audit-noetfield-systems
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: sourcea-brain-chatbot-audit
Source: https://github.com/Noetfield-Systems/SourceA/tree/main/.cursor/skills/sourcea-brain-chatbot-audit
Command: npx skills add https://github.com/Noetfield-Systems/SourceA --skill sourcea-brain-chatbot-audit-noetfield-systems

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Public chatbots drift out of sync with site content, producing stale or wrong answers about pricing, products, and developer paths. This Skill provides a structured audit ritual for the SourceA Brain chatbot so operators can verify knowledge freshness, run quality evals, and catch gaps before shipping worker or landing page changes. ## Core Features & Use Cases - SSOT manifest audit: Reads data/CHATBOT_KNOWLEDGE_MANIFEST.json to check sources, lanes, and stale flags before any Brain worker edit. - Distill/sync pipeline: Runs distill_www_to_brain_knowledge_v1.py, distill_docs_to_brain_knowledge_v1.py, and sync_brain_chat_knowledge_v1.py to rebuild the knowledge bundle from allowlisted sources. - Eval gates: Enforces P0/P1/P2/P3 eval buckets (90%/85% thresholds) via test_brain_chat_quality_v1.py, blocking deploys on failure. - Use Case: Before editing cloud/workers/sourcea-brain-chat-v1/ or updating landing page copy, load this Skill to run the Phase 1 audit ritual, refresh the knowledge bundle, and confirm the worker health endpoint reports the expected knowledge.bundle_version. ## Quick Start Ask the assistant to run the SourceA Brain chatbot audit ritual and check the knowledge manifest, gap report, and worker health before making any chatbot changes.

Frequently Asked Questions about sourcea-brain-chatbot-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit a public chatbot knowledge base for stale answers?

Read the knowledge manifest to check sources and stale flags, review the gap report for known failures, then query the worker health endpoint for knowledge.chunk_count and knowledge.bundle_version. Run the refresh script to distill, sync, and eval the corpus.

How do I run eval tests before deploying a chatbot worker?

Run test_brain_chat_quality_v1.py with --write-report --json to execute the canonical eval buckets. P0 positioning requires 90 percent and P1/P2 buckets require 85 percent; failing P0 blocks the deploy.

When should I load this chatbot audit skill?

Load it before editing the sourcea-brain-chat-v1 worker or sourcea-chatbot.js widget, after landing, pricing, or forge page copy changes, and whenever the founder asks about chat quality, stale answers, or the public knowledge base.

Can I paste raw markdown docs directly into the chatbot prompt?

No. The skill bans dumping raw docs into prompts. Use the manifest allowlist, then run the distill scripts to produce chunks and the sync script to build the knowledge bundle before redeploying the worker.

What content is banned from the public Brain chatbot answers?

Banned content includes invented pricing, custody claims, secrets, OpenRouter or model names, API keys, internal Mac ports, governance jargon like PASS/BLOCK, and leading responses with dollar amounts.