memory-corpus-ingest

Ingest large document corpora into memory as compact map pages with cold-stored raw files.

1.2k|166|Updated Feb 7, 2026
One-click install
npx skills add https://github.com/vellum-ai/vellum-assistant --skill memory-corpus-ingest
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: memory-corpus-ingest
Source: https://github.com/vellum-ai/vellum-assistant/tree/main/skills/memory-corpus-ingest
Command: npx skills add https://github.com/vellum-ai/vellum-assistant --skill memory-corpus-ingest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Large datasets like meeting recording archives, transcript collections, and document dumps are too big to load into an assistant's memory directly, yet users still need them searchable. This Skill ingests such corpora by cold-storing raw files in a workspace imports directory and writing only a small set of map pages into memory, keeping the corpus searchable on demand without bloating memory.

Core Features & Use Cases

  • Cold-store with credential screening: Copies raw files into imports/<source>/ after scanning filenames and contents for secrets, excluding anything sensitive before it lands.
  • Inventory and slice planning: Runs scripts/inventory.ts to census the corpus (file counts, sizes, extensions, date ranges) and propose date-windowed slices sized for one skim pass each.
  • Bounded map generation: Skims each slice into 10-50 map pages following a strict template, ingests them via assistant memory ingest with dry-run validation, and authors a drill-in retrieval skill with runnable search scripts.
  • Use Case: A user exports a year of Fathom meeting recordings. The Skill lands the transcripts in the cold store, builds quarterly map pages capturing decisions and open threads, and creates a fathom-lookup skill so questions like "what did we decide in the March planning call" route to the right files.

Quick Start

Ask the assistant to import your Fathom recordings export folder into memory so it stays searchable without pasting everything into chat.

Frequently Asked Questions about memory-corpus-ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I import a large document archive into an AI assistant's memory?

Cold-store the raw files under a workspace imports directory, census them into date-windowed slices, skim each slice into compact map pages, and ingest only the map via the memory ingest CLI. Raw content never enters memory; a drill-in retrieval skill reaches the files on demand.

How do I make a Fathom recordings export searchable by an assistant?

Land the export under imports/fathom, extract meeting dates from filenames or metadata, slice monthly or quarterly, and write one map page per slice capturing decisions, recurring topics, and open threads. A scoped ripgrep search script with date filtering handles drill-in queries.

Does corpus ingest handle credentials or secrets in the source files?

Yes. Before copying, the workflow screens filenames and file contents for credential patterns like API keys, tokens, and private keys using ripgrep with hidden-file scanning. Flagged files are excluded from the cold store or resolved with the user first.

How many memory pages does a large corpus ingest create?

The map stays between roughly 10 and 50 pages regardless of corpus size. If the corpus grows, slices get coarser (month to quarter to year) or pages get denser; the page count does not scale with file count.

When should I not use corpus ingest for importing data?

Avoid it for small sets of distilled memories or exports from another assistant, which fit a migration skill instead, and for ChatGPT conversation history, which has a dedicated import path. It is designed for corpora too large to hold in memory directly.