skills.rest
frederico-kluser
frederico-kluserCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

running-and-verifying-crawls

Runs, smoke-tests, and verifies the newsletter crawler end to end via npm commands.

2|1|Updated Jun 29, 2026
Search Tags:#testing#npm#verification#sqlite#deploy#crawler#smoke test
One-click install
npx skills add https://github.com/frederico-kluser/newsletter-crawler --skill running-and-verifying-crawls-frederico-kluser
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: running-and-verifying-crawls
Source: https://github.com/frederico-kluser/newsletter-crawler/tree/main/.agents/skills/running-and-verifying-crawls
Command: npx skills add https://github.com/frederico-kluser/newsletter-crawler --skill running-and-verifying-crawls-frederico-kluser

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? After changing the newsletter crawler, you need a reliable way to prove the change works without burning LLM tokens or risking the user's real archive. This Skill provides the exact command sequence to smoke-test, run bounded crawls, verify cache behavior, export results, and validate the deploy pipeline. ## Core Features & Use Cases - Smoke and unit testing: npm run status boots the SQLite schema and imports every module; npm test runs 868+ tests with strict NC_HOME isolation so the user's real database is never touched. - Bounded crawl verification: npm run crawl -- --max-pages 1 --max-articles 3 exercises fetch, LLM, and extraction cheaply, and re-running proves the selector cache hit and resumability. - Export, deploy, and recovery: covers markdown/web export with secret redaction, the one-command npm run deploy flow with live-site polling, and restoring the archive from git history via ncrawl restore. - Use Case: You fixed a date-parsing bug and want to confirm it end to end — run the unit tests, a bounded crawl, inspect status counts, and export, all without touching production data. ## Quick Start Ask the assistant to run a bounded test crawl with npm run crawl using --max-pages 1 and --max-articles 3, then check the results with npm run status.

Frequently Asked Questions about running-and-verifying-crawls

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a bounded test crawl without high LLM cost?▼

Run npm run crawl -- --max-pages 1 --max-articles 3 to exercise fetch, LLM, and extraction with minimal token usage. Re-running the same command should show a selector cache hit with zero additional LLM cost.

How do I verify my crawler change works end to end?▼

Start with npm run status to smoke-test imports and the SQLite schema, then npm test for the unit suite, then a bounded crawl. Finish with npm run export -- --format md and inspect status counts to confirm articles were saved.

How do I check crawl status and database counts?▼

Run npm run status to see sources, pages, articles, selectors, and frontier counts. For detail, open the SQLite database read-only with better-sqlite3 and query the tables directly.

Can I reset the crawler database safely?▼

Yes, use npm run reset -- --yes --confirm <article count>, which creates a backup first and aborts if the confirmation number is wrong. Never run destructive commands against the real NC_HOME for testing; use a sandbox with a temp NC_HOME instead.

Why does the crawler return 401 errors from OpenRouter?▼

A 401 usually means a stale OPENROUTER_API_KEY in your shell environment overriding the .env file. Verify the key with curl to https://openrouter.ai/api/v1/key using the Authorization Bearer header; a 200 response means the key is valid.

How do I restore the article archive after wiping the database?▼

Run ncrawl restore to rebuild the SQLite database from the versioned snapshots in webapp/public/data git history. An empty database with git history also triggers automatic bootstrap restore on the first useful command.

Related Skills

Explore More Software Engineering Skills→
esimplicityincesimplicityinc

katalyst-bdd-quickstart

Bootstrap a Katalyst BDD test project with stack-tests and npm/yarn commands.

Official
Intermediate
EmasoftEmasoft

amvcp-share-pages

Deploy a self-contained HTML page to Vercel and return a public live URL.

Community
Intermediate
tony-admtony-adm

cicd-diagnostics

Diagnose DotCMS GitHub Actions CI/CD failures with log analysis and root-cause classification.

Community
Advanced
dennislee928dennislee928
54

firecrawl-scraper

Scrapes and crawls websites into LLM-ready markdown or JSON.

Community
Advanced