skills.rest
dpla
dplaOFFICIAL
Explore all skills by this author →
WebsiteGitHub

DPLA Hub Ingest

Automate DPLA hub ingest from harvest to S3 sync on EC2.

35|12|Updated Jan 10, 2017
Search Tags:#etl#mapping#jsonl#enrichment#harvest#dpla#hub-ingest
One-click install
npx skills add https://github.com/dpla/ingestion3 --skill dpla-hub-ingest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: DPLA Hub Ingest
Source: https://github.com/dpla/ingestion3/tree/main/.cursor/skills/dpla-hub-ingest
Command: npx skills add https://github.com/dpla/ingestion3 --skill dpla-hub-ingest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the end-to-end DPLA hub ingestion pipeline on the ingest EC2 instance, handling harvest, mapping, enrichment, JSONL packaging, and S3 sync to publish data for indexing.

Core Features & Use Cases

  • Orchestrates the complete hub ingest workflow from harvest to JSONL and S3 sync on a single EC2 instance.
  • Coordinates harvest, mapping, enrichment, and JSONL export with integrated checks, safety prompts, and Slack notifications.
  • Supports hub-specific configurations via i3.conf and resumable execution using ingest.sh for recoverable failures.

Quick Start

Run the hub ingest on the ingest EC2 instance by specifying the hub name.

Frequently Asked Questions about DPLA Hub Ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate the full DPLA hub ingest workflow from harvest to S3 sync?▼

You can automate the DPLA hub ingest workflow by orchestrating harvest, mapping, enrichment, JSONL packaging, and S3 sync on a single ingest EC2 instance using hub-specific configurations defined in i3.conf.

What is the process for packaging enriched DPLA metadata into JSONL for indexing?▼

The process involves harvesting data, mapping fields, and applying enrichment before exporting the results into JSONL format and syncing them to S3 for downstream indexing.

Can I resume a failed DPLA hub ingest job on an EC2 instance without starting over?▼

Yes, you can resume a failed DPLA hub ingest job by using the ingest.sh script, which supports recoverable execution so you can continue the pipeline from the point of failure.

Does the DPLA hub ingest pipeline support Slack notifications for progress and failures?▼

Yes, the DPLA hub ingest pipeline integrates Slack notifications to alert you about both workflow progress and failures during the harvest, mapping, and enrichment stages.

How do I configure hub-specific settings for a DPLA ingest run on EC2?▼

You configure hub-specific settings for a DPLA ingest run by editing the i3.conf file, which allows the pipeline to apply the correct parameters for harvesting and mapping your designated hub.

What are the limitations of running the entire DPLA hub ingest pipeline on a single EC2 instance?▼

Running the full DPLA hub ingest on a single EC2 instance coordinates all stages sequentially with safety prompts, meaning recoverable failures will pause the workflow and require manual intervention via ingest.sh to resume.

Related Skills

Explore More Software Engineering Skills→
dpladpla
35

dpla-community-webs-ingest

Automate Community Webs data harvesting and ingestion into the DPLA pipeline via EC2 and SSM.

Official
Intermediate
dpladpla
35

dpla-staged-report

Identify hubs with newly staged JSONL data in S3 for a given month.

Official
Intermediate
dpladpla
35

dpla-orchestrator

Run the DPLA ingest orchestrator for harvest, mapping, enrichment, JSONL export, anomaly detection, and S3 sync.

Official
Advanced
MarioDeFelipeMarioDeFelipe
25

datasphere-data-flows

Orchestrate SAP Datasphere replication, transformation, and task chain flows.

Community
Advanced