firecrawl-knowledge-ingest

Use Firecrawl browser navigation to handle authentication flows and render JS-heavy pages that static scrapers cannot access. The workflow opens the portal, navigates authenticated sections, and scrapes article content as markdown while respecting authentication boundaries.

Firecrawl browser follows sidebar navigation, next links, pagination, and load-more controls to discover all article URLs. Each article is then scraped as markdown with metadata like title, section, and last updated date.

Yes, Firecrawl browser renders JavaScript content, making it suitable for JS-heavy docs portals where plain HTTP fetching returns incomplete pages. For public URLs, Firecrawl map can supplement discovery of article links.

Output is JSON or markdown containing source, URL, extraction timestamp, total article count, and sections array. Each article includes title, URL, section, content, and metadata fields, with code examples and tables preserved.

Pages fail due to access restrictions, loading errors, or authentication boundaries. The workflow tracks extraction progress and page failures, reporting them in a Failed Or Restricted Pages section of the final deliverable.