dpla-orchestrator

To coordinate a multi-stage data ingestion pipeline, this orchestrator manages harvest, mapping, enrichment, JSONL export, anomaly detection, and S3 sync across one or more hubs with parallel execution and per-hub status tracking.

Before running the DPLA ingest orchestrator, you need to activate a Python virtual environment and configure environment variables in a .env file to ensure the scheduler.orchestrator.main entry point executes properly.

Yes, you can test pipeline ingestion using the supported dry-run mode. The orchestrator allows dry-run execution to validate the harvest and mapping workflow across environments without applying permanent changes.

You monitor pipeline ingestion status through per-hub monitoring logs and error escalation. The orchestrator provides per-hub completion reporting and Slack notifications to ensure reliable operations and track failures.

The best way to retry failed data harvest and enrichment tasks is using the orchestrator's built-in retry execution capability, which coordinates reprocessing across hubs while maintaining auditable run logs.

Using a Python venv orchestrator for S3 sync and JSONL export ensures an isolated, consistent environment for executing the multi-stage ingest, reducing manual coordination and preventing dependency conflicts.