ingest-agent

To automate external content ingestion, you provide a content URL to fetch articles, posts, videos, or repositories. The system then extracts metadata, cleans the content, and stores it with unique file names in structured knowledge base directories.

Extracting metadata from articles and websites is handled automatically by parsing HTML with BeautifulSoup and lxml. The skill identifies content types, pulls titles, authors, and dates, and formats them for structured storage in your knowledge base.

Organizing downloaded web content by type is achieved by generating unique file names and routing content to appropriate directories based on whether it is an article, post, video, paper, or repository. The skill also updates an index file to track everything.

Yes, you need Python and BeautifulSoup along with requests, lxml, and yaml to ingest external content. These libraries handle the HTTP fetching, HTML parsing, and configuration required for content extraction and metadata processing.

Supported content types for automated ingestion and cleanup include articles, posts, videos, papers, repositories, and general websites. The skill processes each type, extracts relevant metadata, and formats the output for structured storage.

Yes, you can use this to ingest GitHub repositories and research papers. The skill fetches external content from URLs, automatically extracts specific metadata for each content type, and organizes the files into your knowledge base.