skills.rest
ngochuy13
ngochuy13COMMUNITY
Explore all skills by this author →
GitHub

archive-crawler

Scan personal file archives and ingest high-signal content into structured brain pages.

Updated Apr 23, 2026
Search Tags:#archive#document ingestion#personal knowledge#mbox#content triage#gmail-takeout#allow-list safety
One-click install
npx skills add https://github.com/ngochuy13/intern-dev --skill archive-crawler-ngochuy13
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: archive-crawler
Source: https://github.com/ngochuy13/intern-dev/tree/main/skills/archive-crawler
Command: npx skills add https://github.com/ngochuy13/intern-dev --skill archive-crawler-ngochuy13

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It solves the problem of turning scattered personal files across local folders, cloud exports, and mailbox archives into a curated set of “high-signal” brain pages without drowning you in noise.

Core Features & Use Cases

  • Gold-focused archive crawling: inventories a filesystem or archive tree, triages items, and surfaces likely personal writing, ideas, relationships, and origin-story material while skipping obvious noise.
  • Interactive review with manifest tracking: creates and updates a per-archive manifest at projects/<archive-slug>/STATUS.md to prevent re-showing items and to record exact user reactions.
  • Safety gate via allow-list: refuses to run unless gbrain.yml explicitly sets archive-crawler.scan_paths to an allow-list of permitted locations.
  • Schema-generic ingestion routing: applies filing rules read at runtime from _brain-filing-rules.json so content lands under the right sanctioned directories (e.g., originals/, personal/, ideas/, people/).

Quick Start

Ask the AI to run archive-crawler using only the allow-listed paths configured in your gbrain.yml, and then review the proposed priority queue before crawling.

Frequently Asked Questions about archive-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ingest Gmail takeout mbox archives into structured knowledge pages?▼

To ingest Gmail takeout mbox archives, you must configure an explicit allow-list in your `gbrain.yml` file setting `archive-crawler.scan_paths` to permitted locations, then the crawler triages mailbox exports and routes high-signal content into structured brain pages based on filing rules.

What is the best way to scan personal file archives for high-signal content without noise?▼

The best way to scan personal file archives is using a gold-focused crawling approach that inventories filesystem trees, triages items to surface personal writing and ideas, and skips obvious noise by applying a gold filter to route only high-signal content into sanctioned directories.

Can I crawl Dropbox trees and Backblaze B2 prefixes for personal knowledge ingestion?▼

Yes, you can crawl Dropbox trees and Backblaze B2 prefixes by mounting them as local directory paths and adding them to the `archive-crawler.scan_paths` allow-list in `gbrain.yml`, enabling the crawler to inventory and triage files across those cloud export formats.

How does archive triage prevent re-processing files I have already reviewed?▼

Archive triage prevents re-processing by creating and updating a per-archive manifest at `projects/<archive-slug>/STATUS.md` that tracks exact user reactions and ensures previously reviewed items are not surfaced again during subsequent crawling sessions.

Do I need to configure an allow-list before scanning local directory mounts and mbox files?▼

Yes, you must configure an allow-list before scanning; the safety gate refuses to run unless `gbrain.yml` explicitly sets `archive-crawler.scan_paths` to permitted locations, ensuring only authorized local directory mounts and mbox files are crawled.

How are ingested archive pages routed without hardcoding directory layouts?▼

Ingested archive pages are routed by applying filing rules read at runtime from `_brain-filing-rules.json`, directing content into sanctioned directories like `originals/`, `personal/`, `ideas/`, and `people/` without hardcoding era-specific layouts.

Related Skills

Explore More Personal & Entrepreneur Skills→
weipingweiping
1

archive-crawler

Crawl allow-listed personal archives and convert high-signal content into brain pages.

Community
Advanced
inbrainfuninbrainfun
174

archive-crawler

Crawl approved archive paths and generate structured manifests for content ingestion.

Community
Advanced
Morris-UTRUSTMorris-UTRUST

voice-note-ingest

Ingest voice notes and route verbatim transcripts to brain pages.

Community
Advanced
cas-bigdatalabcas-bigdatalab
541

batch_collector

Scan local files or directories and generate a file inventory with metadata.

Official
Intermediate