Crawl entire websites by following links from a starting URL.
npx skills add https://github.com/olostep/olostep-cursor-plugin --skill crawl-olostep
Please help me install this Agent Skill. Skill: crawl Source: https://github.com/olostep/olostep-cursor-plugin/tree/main/skills/crawl Command: npx skills add https://github.com/olostep/olostep-cursor-plugin --skill crawl-olostep
This Skill automates the tedious and incomplete process of manually collecting pages from multi-page websites, providing comprehensive context for AI tasks.
Crawl the website starting at https://example.com and process up to 10 pages.
High-intent search queries and answers about installing and using this skill.
You can automate website ingestion by using a crawling tool that follows internal links from a starting URL, rendering pages in a full browser to collect documentation and knowledge bases for AI context.
The best way is using a full browser rendering approach with anti-bot bypass, which successfully scrapes JavaScript-heavy and bot-protected documentation sites by following internal links automatically.
Yes, you can export scraped website content as markdown, HTML, JSON, or text. The ingestion process supports these various output formats to prepare data for your specific AI context requirements.
Yes, web scraping supports geo-targeting options to render and collect localized website content, ensuring the ingested pages reflect specific geographic locations during the crawl.
You can collect multiple pages from a knowledge base automatically by using a tool with automated link following, which discovers and processes internal pages from a starting URL up to your specified limit.
You need full browser rendering for web scraping when targeting JavaScript-heavy pages or bot-protected sites, ensuring dynamically loaded content is fully rendered and captured during the website ingestion process.
jscraikInitiate, monitor, and export website crawls via the Cloudflare Browser Rendering API.
SethyPagnaStore embeddings and metadata in Chroma collections for semantic search.
dnsnamdangGenerate test case documents from requirements or design specifications.
DaddyElonMusk69Scrape JavaScript-rendered and Cloudflare-protected web pages via CLI or Python.
