searching-the-corpus

Use the search command with Mode A for exhaustive per-article Flash relevance judging, or Mode B for cheap tag-based retrieval using 5 Pro calls mapped to the taxonomy facets. The web UI offers soft batched judging and a deep per-article toggle.

Mode A scans every article in scope with Flash relevance judging returning direct/similar/none verdicts, with a cost confirmation guard. Mode B makes exactly 5 Pro calls to derive facet tags, then retrieves articles via json_each tag matching ranked by match count.

The summarize stage writes title_pt and summary_pt columns using a Flash high model, keeping the original content untouched for search and tags. It is idempotent via a summary_pt IS NULL check and runs automatically as a post-crawl hook.

Yes, GET /api/search/stream emits SSE progress, hit, done, and error events at article level in both soft and deep modes. The client uses an abortable fetch-based parser rather than EventSource so it can read 428/409 status codes and cancel.

The static webapp contains verbatim copies of the relevance schemas, batch prompts, and merge logic from src/llm.js and src/search.js because the browser cannot import Node modules. Editing the rubric without updating webapp/src/lib/search.js desynchronizes browser-side AI search.

Never rebuild it: the 9-facet taxonomy already exists in config/taxonomy.json with classification persisted to article_tags. Search features should reuse validateFacetTags, RETRIEVAL_FACETS, and isToolByTags instead of creating new classification logic.