skills.rest
webdevarif
webdevarifCOMMUNITY
Explore all skills by this author →
Website𝕏@webdevarifGitHub

firecrawl-knowledge-base

Builds LLM-ready knowledge bases from web content using Firecrawl scraping and search.

Updated Apr 18, 2026
Search Tags:#documentation#markdown#rag#firecrawl#knowledge base#fine-tuning#web scraping
One-click install
npx skills add https://github.com/webdevarif/claude-skills --skill firecrawl-knowledge-base-webdevarif
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: firecrawl-knowledge-base
Source: https://github.com/webdevarif/claude-skills/tree/main/firecrawl-knowledge-base
Command: npx skills add https://github.com/webdevarif/claude-skills --skill firecrawl-knowledge-base-webdevarif

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Turning scattered web documentation and topic content into organized, LLM-ready markdown requires manual scraping, cleaning, and structuring. This Skill automates that pipeline using Firecrawl to produce reference docs, RAG chunks, training datasets, or documentation mirrors. ## Core Features & Use Cases - Web Content Collection: Uses Firecrawl map, search, and scrape to gather documentation pages and topic-based corpora into markdown. - Multiple Output Modes: Produces reference markdown with sources.json, RAG chunks with manifest.json, training data as training-data.jsonl, or full documentation mirrors. - Parallel Collection: Supports splitting work across sub-agents by docs section, source type, or pipeline stage. - Use Case: Point it at a framework's documentation site and receive a structured .firecrawl/ directory of clean markdown files with preserved code examples, ready for RAG ingestion or fine-tuning. ## Quick Start Ask the assistant to build a RAG-ready knowledge base from a documentation URL using your Firecrawl API key.

Frequently Asked Questions about firecrawl-knowledge-base

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a knowledge base from web documentation?▼

Provide a documentation URL or topic and a goal such as reference, RAG, training, or docs mirror. The skill uses Firecrawl map and scrape to collect pages into organized markdown under a .firecrawl/ directory with source metadata.

How to create RAG-ready chunks from website content?▼

Select the RAG output mode, which scrapes pages into markdown and generates chunk files plus a manifest.json. Code examples and tables are preserved, and boilerplate navigation is removed where possible.

Does Firecrawl knowledge base require an API key?▼

Yes, a FIRECRAWL_API_KEY environment variable is required for hosted Firecrawl requests. Without it, the map, search, and scrape operations cannot run.

Can I generate fine-tuning datasets from scraped web pages?▼

Yes, the training output mode produces training-data.jsonl and training-metadata.json from scraped source files. You may be asked which training format you want before generation begins.

What output formats does web scraping to markdown support?▼

Four modes are supported: reference markdown with sources.json, RAG chunks with manifest.json, training JSONL with metadata, and a complete documentation mirror with a table of contents.

Related Skills

Explore More Software Engineering Skills→
ironkid90ironkid90

firecrawl-knowledge-base

Collect and structure web content from URLs into markdown documents.

Community
Advanced
kreuzberg-devkreuzberg-dev
9.2k

chunking

Split extracted text into sized chunks for LLM context windows and RAG ingestion.

Official
Intermediate
vecearvecear

doc-to-vector-dataset-generator

Convert documents into chunked JSONL datasets with metadata for embedding search.

Community
Advanced
coseto6125coseto6125

seo-firecrawl

Crawl, map, scrape, and search websites via the Firecrawl MCP server for SEO analysis.

Community
Intermediate