Crawl websites and save pages as markdown files using Python scripts.
npx skills add https://github.com/tony2015116/openclaw-backup --skill crawl-tony2015116
Please help me install this Agent Skill. Skill: crawl Source: https://github.com/tony2015116/openclaw-backup/tree/main/skills/tavily/crawl Command: npx skills add https://github.com/tony2015116/openclaw-backup --skill crawl-tony2015116
This Skill simplifies the process of crawling websites and saving pages as local markdown files, enabling offline access or analysis of web content without coding.
Use the crawl skill to crawl the website 'https://docs.example.com' and save the results to a local directory.
High-intent search queries and answers about installing and using this skill.
To crawl a website and save pages as markdown files, this skill uses Python scripts to scrape web content and export each page locally. It requires authentication via the Tavily MCP server OAuth or an API key to retrieve and convert the pages.
The best way to scrape web content for offline analysis is using a tool that downloads multiple pages and exports them as local markdown files. This approach supports depth and breadth limits alongside path filtering for targeted document retrieval.
You need either an API key or OAuth authentication through the Tavily MCP server to perform web scraping. Both methods provide secure access for crawling websites and retrieving documentation or knowledge bases without direct coding.
You can set depth and breadth limits for a website crawl to control how many pages are scraped. The skill also supports path filtering and semantic instructions, ensuring you only retrieve targeted offline content.
This web scraping tool works with BeautifulSoup and lxml as its core dependencies for parsing HTML. Combined with the requests library, it navigates websites and saves the extracted content as structured markdown files.
You should use path filtering and semantic instructions for document retrieval when you need to target specific sections of a large website. This prevents crawling irrelevant pages and ensures only the desired offline content is saved as markdown.
moongci38-ossAnalyze website structure, components, and APIs to generate a reimplementation guide.
joonlabConvert web page DOM content into Markdown files with YAML frontmatter.
aivos-xieParse Markdown documents into a structured knowledge base for cross-referencing and queries.
ddalcuExecute JavaScript, shell commands, and headless browser actions in a secure sandbox.
