skills.rest
dpla
dplaOFFICIAL
Explore all skills by this author →
WebsiteGitHub

dpla-oai-harvest-watch

Parse OAI harvest logs to report set-by-set progress and ETA.

35|12|Updated Jan 10, 2017
Search Tags:#logs#progress#watch#harvest#eta#sets#oai
One-click install
npx skills add https://github.com/dpla/ingestion3 --skill dpla-oai-harvest-watch
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dpla-oai-harvest-watch
Source: https://github.com/dpla/ingestion3/tree/main/.cursor/skills/dpla-oai-harvest-watch
Command: npx skills add https://github.com/dpla/ingestion3 --skill dpla-oai-harvest-watch

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Watching and coordinating long-running OAI harvests is error-prone and hard to gauge progress. This Skill provides a clear, real-time view of set-by-set progress and estimated completion times by parsing harvest logs for set transitions.

Core Features & Use Cases

  • Real-time log watching to detect ListRecords and per-set transitions.
  • ETA estimation for ongoing harvests using harvest.setlist totals or manual totals.
  • Useful for hubs with long-running ingests to update planning and status reports.

Quick Start

Run a harvest, capture its log, and run the watcher against the log to see per-set progress and ETA.

Frequently Asked Questions about dpla-oai-harvest-watch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor OAI harvest progress for each set?▼

Estimate OAI harvest completion times by parsing harvest logs for set transitions and comparing them against configurable totals from harvest.setlist, i3.conf, or manually provided values to calculate an ETA.

Why is tracking long-running OAI ingests difficult to gauge?▼

Long-running OAI ingests are difficult to gauge because set transitions occur over extended periods, making it error-prone to manually track progress without a deterministic log parser to detect ListRecords and per-set completion.

Can I use manual totals to estimate OAI harvest ETA?▼

Yes, you can use manual totals to estimate OAI harvest ETA by configuring them in i3.conf, allowing the watcher to calculate progress estimates even without an automated harvest.setlist.

What's the best way to watch OAI harvest logs for set-by-set status?▼

The best way to watch OAI harvest logs is using a deterministic log parsing script like watch-oai-harvest.py, which detects set transitions and provides ongoing progress estimates for long-running ingests.

Do I need harvest.setlist to monitor OAI ingest progress?▼

No, you do not need harvest.setlist to monitor OAI ingest progress; the watcher supports configurable totals from i3.conf or manual totals to provide set-by-set progress and ETA.

Related Skills

Explore More Software Engineering Skills→
dpladpla
35

dpla-run-ingest

Automate hub ingest runs by selecting runbooks and verifying output artifacts.

Official
Intermediate
dpladpla
35

dpla-community-webs-ingest

Automate Community Webs data harvesting and ingestion into the DPLA pipeline via EC2 and SSM.

Official
Intermediate
interflownexinterflownex
2

bigquery-data-transfer-service

List and inspect BigQuery Data Transfer Service configurations across GCP projects.

Community
Advanced
NarubyRiverlioneNarubyRiverlione

task-completion-report

Parse tasks.md to compute per-phase and overall completion percentages.

Community
Basic