dpla-ingest-debug

Diagnose and fix DPLA hub ingestion failures across pipeline stages.

35|12|Updated Jan 10, 2017

Search Tags:#debugging #logs #ingestion #orchestrator #harvest #remap #s3-sync

Ready to run

npx skills add https://github.com/dpla/ingestion3 --skill dpla-ingest-debug

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: dpla-ingest-debug
Source: https://github.com/dpla/ingestion3/tree/main/.cursor/skills/dpla-ingest-debug
Command: npx skills add https://github.com/dpla/ingestion3 --skill dpla-ingest-debug

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Debug and fix DPLA hub ingestion failures across harvest, mapping, enrichment, jsonl, s3-sync, and anomaly. Use when a hub fails, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.

Core Features & Use Cases

Gather context and logs to identify where the pipeline failed.
Determine the correct runbook and rerun only the minimal steps (harvest, remap, s3-sync) for a hub.
Verify outputs via _SUCCESS markers and provide escalation-ready diagnostics.

Quick Start

Follow the fast path: identify the hub and stage from logs and run the minimal steps (harvest, remap, s3-sync) for that hub.

Frequently Asked Questions about dpla-ingest-debug

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I debug DPLA hub ingestion failures across harvest and s3-sync stages?▼

Debug DPLA hub ingestion failures by gathering context and logs to identify the failed pipeline stage, then running the minimal steps for harvest, remap, and s3-sync. This isolates the issue and enables targeted re-runs for the specific hub.

What's the best way to retry a failed DPLA hub stage without rerunning the entire pipeline?▼

Retry a failed DPLA hub stage by identifying the failing step from logs, selecting the correct runbook, and executing only the minimal required script like harvest.sh, remap.sh, or s3-sync.sh for that individual hub.

Why does DPLA ingestion fail during the mapping or enrichment workflow?▼

DPLA ingestion fails during mapping or enrichment due to data anomalies or processing errors in the pipeline. Diagnose these failures by checking escalation reports and logs to pinpoint the exact stage causing the breakdown.

Do I need environment setup to diagnose DPLA orchestrator failures?▼

You need environment setup to diagnose DPLA orchestrator failures, including sourcing .env or i3_conf files. Access to data/escalations, logs, and runbooks is required to run harvest, remap, and s3-sync scripts.

How do I verify outputs after rerunning a DPLA harvest or remap workflow?▼

Verify outputs after rerunning a DPLA harvest or remap workflow by checking for _SUCCESS markers. These markers confirm the process completed correctly and provide escalation-ready diagnostics for the hub.

When should I not use a minimal re-run approach for DPLA ingestion debugging?▼

Avoid a minimal re-run approach for DPLA ingestion debugging when the failure spans multiple interconnected stages or requires a full pipeline reset. Use it only for isolated stage failures like single harvest, remap, or s3-sync issues.