skills.rest
frederico-kluser
frederico-kluserCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

extending-the-crawler

Guides adding sources, modules, CLI commands, and fetch strategies to the newsletter crawler.

2|1|Updated Jun 29, 2026
Search Tags:#newsletter#cli#extension#web scraping#node.js#source management#crawler
One-click install
npx skills add https://github.com/frederico-kluser/newsletter-crawler --skill extending-the-crawler-frederico-kluser
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: extending-the-crawler
Source: https://github.com/frederico-kluser/newsletter-crawler/tree/main/.agents/skills/extending-the-crawler
Command: npx skills add https://github.com/frederico-kluser/newsletter-crawler --skill extending-the-crawler-frederico-kluser

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Developers extending the newsletter crawler risk editing the wrong config file, breaking conventions, or silently no-oping commands because the project has two sources.json files, strict module placement rules, and a specific CLI wiring pattern. ## Core Features & Use Cases - Source management: Add or replace newsletter sources via the add command or UI wizard, understanding that NC_HOME/sources.json is live while config/sources.json is only the install seed, with AI-detected source types (index vs listing). - Subset crawling: Crawl specific sources with --sources/--source/--only flags using the filterSeedSources helper with explicit precedence and warnings. - Extension recipes: Step-by-step conventions for adding src/ modules, CLI commands (commands.js + index.js dispatch + npm script + UI screen), publications, and fetch/extract strategies behind fetchSmart/clean.js. - Use Case: You want to add a new weekly newsletter to the crawler. Follow the procedure to run npm run add, let detectSourceType classify it, then run a bounded crawl with --since to confirm links are found and a selector is cached. ## Quick Start Ask the assistant to add a new newsletter source to the crawler and verify it with a bounded crawl using --since and --max-articles.

Frequently Asked Questions about extending-the-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new newsletter source to the crawler?▼

Run npm run add or use the UI wizard, which persists the source to the live NC_HOME/sources.json via addSourceToConfig. The source type (index vs listing) is auto-detected by AI through detectSourceType, or forced manually with --type. Then run a bounded crawl to confirm links are found.

How do I crawl only specific sources instead of all of them?▼

Use the --sources flag with a comma-separated list of exact names or normalized URLs, which takes precedence over --source and --only. The filterSeedSources helper warns on unmatched items so the run never silently no-ops.

Why did editing config/sources.json not change my crawler sources?▼

The repo's config/sources.json is only a seed copied once on fresh install. The live file is NC_HOME/sources.json, so existing installations keep their own copy and you must edit both to change the factory set.

How do I add a new CLI command to a Node.js crawler project?▼

Put the command body in src/commands.js as an exported side-effect-free function, wire it into the dispatch in src/index.js ending with db.close(), add an npm script in package.json, and optionally add a screen in src/ui/screens.js plus a route in src/ui/App.js.

What is the difference between purge and remove for a crawler source?▼

Purge deletes collected data but keeps the source row, so the seed can bring it back. The remove command with --yes fully unregisters the source: it deletes data, searches, the sources table row, and removes it from sources.json.

Related Skills

Explore More Software Engineering Skills→
CommonCapitalCommonCapital
2

nodejs-sandbox

Execute Node.js and TypeScript code in a Docker container.

Community
Intermediate
73ai73ai
4

web-search

Search the web and return JSON results from multiple search engines.

Official
Intermediate
sail-sailsail-sail
14

create-module

Scaffold business modules with SQL and TS templates and register them in tables.ts.

Community
Intermediate
fgarofalo56fgarofalo56
6

scrapy

Build custom Scrapy spiders to crawl websites and extract data.

Community
Advanced