bulk-ingestion

Orchestrates large-scale data ingestion into brain pages using a durable JSON manifest.

Updated Aug 17, 2026
One-click install
npx skills add https://github.com/AMC-JTC/gbrain-1 --skill bulk-ingestion-amc-jtc
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: bulk-ingestion
Source: https://github.com/AMC-JTC/gbrain-1/tree/main/plugin/skills/bulk-ingestion
Command: npx skills add https://github.com/AMC-JTC/gbrain-1 --skill bulk-ingestion-amc-jtc

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Bulk-ingesting large data sources (audio libraries, email takeouts, document corpora, chat exports) into a knowledge brain fails when progress lives in agent memory, trials are skipped, or crashes force restarts from scratch. This Skill provides a full lifecycle discipline so large ingestion jobs survive crashes, session boundaries, and multi-worker fan-out. ## Core Features & Use Cases - Ten-phase lifecycle: SCHEMA → ACCESS → TRIAL → EVALUATE → IMPROVE → CODIFY → TEST → SKILLIFY → BULK → MONITOR, with a mandatory 5-10 diverse trial loop before any bulk run. - Durable manifest state: a manifest.json source of truth (plus rendered MANIFEST.md) built from authoritative source enumeration, with status derived from artifacts on disk rather than assertions. - Idempotent multi-worker execution: disjoint shard partitioning, checkpointing, per-worker progress files, and verification of outputs on disk instead of trusting subagent success claims. - Use Case: Ingest a 400K-message email takeout into brain pages by defining a page schema, trialing 10 diverse threads, codifying the deterministic transform, then fanning out sharded workers whose progress is tracked and resumable via the manifest. ## Quick Start Ask the agent to bulk ingest all of a chosen data source into the brain and have it build a resumable manifest before processing anything.

Frequently Asked Questions about bulk-ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I bulk ingest a large data source into a knowledge base?

Define a page schema first (template, filing rules, entity propagation, dedup key), then enumerate the source into a manifest, trial 5-10 diverse examples, and only run the full set after the trials pass review. Progress is tracked in the manifest so crashes resume from ground truth.

How do I track progress of a large ingestion across sessions and workers?

Use a two-file manifest: a machine-updatable manifest.json as the source of truth and a rendered MANIFEST.md for human review. Status is derived from output artifacts on disk, and workers process disjoint shards with idempotent restarts and frequent checkpoints.

When should I use bulk ingestion instead of single-item ingest?

Use bulk ingestion for any enumerable set of more than about 20 items, or any job spanning multiple sessions or workers. A single item, such as one meeting note or article, should go through the single-item ingest router instead.

Can multiple workers process the same ingestion manifest in parallel?

Yes, but the manifest has no atomic claim, so the work-list must be partitioned into disjoint shards up front by group or offset range. High-parallelism runs use per-worker progress files that a merge step folds into the master manifest.

Why did my ingestion job report done when most items were never processed?

This happens when completion is declared by inspecting the output folder or trusting a subagent's success claim. The fix is to rebuild state from ground truth: re-scan the source, match outputs by stored backlink, and verify artifacts exist on disk before advancing status.