Discover all URLs on a website using the tvly CLI with filter options.
npx skills add https://github.com/BEIRUX/agent-skills --skill tavily-map-beirux
Please help me install this Agent Skill. Skill: tavily-map Source: https://github.com/BEIRUX/agent-skills/tree/main/tavily-map Command: npx skills add https://github.com/BEIRUX/agent-skills --skill tavily-map-beirux
Quickly discover and list every URL on a website without fetching or extracting page content, so you can locate a specific page or understand site structure without performing a full crawl.
Map all URLs on https://docs.example.com and return the results as JSON using the tvly map command.
High-intent search queries and answers about installing and using this skill.
To discover all URLs on a website without extracting page content, use a site mapping command that returns link inventories only. This approach skips page fetching, making URL discovery faster and less resource-intensive than a full web crawl.
The best way to find a buried page in large documentation sites is using URL discovery with natural language instructions and regex path filters. This targets relevant links without crawling entire pages, quickly locating the specific documentation you need.
Yes, you can filter discovered URLs using domain regex and path pattern filters. This allows you to limit results to specific blog or API paths, targeting relevant links before performing any extraction or deeper scraping.
Yes, you need the tvly CLI installed and available on your PATH to perform URL discovery. This command-line tool executes the site mapping process and supports options for depth limits, breadth limits, and JSON output.
You can limit the depth and breadth of a site mapping crawl by passing specific options to the URL discovery command. These parameters control how many links to follow and how deep the traversal goes, constraining the resulting site inventory.
a2ajinkyaDiscover website URLs with Tavily CLI filters and JSON output.
refly-aiExtract content from URLs and perform site-specific searches with Jina.
while-basicExtract web page content as Markdown or structured data via the Firecrawl API.
dennislee928Scrapes and crawls websites into LLM-ready markdown or JSON.