skills.rest
dpla
dplaOFFICIAL
Explore all skills by this author →
WebsiteGitHub

dpla-s3-ops

Sync hub data to S3 and verify JSONL exports using the dpla profile.

35|12|Updated Jan 10, 2017
Search Tags:#aws#synchronization#data#ingestion#s3#jsonl#hub
One-click install
npx skills add https://github.com/dpla/ingestion3 --skill dpla-s3-ops
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dpla-s3-ops
Source: https://github.com/dpla/ingestion3/tree/main/.cursor/skills/dpla-s3-ops
Command: npx skills add https://github.com/dpla/ingestion3 --skill dpla-s3-ops

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

The skill streamlines AWS S3 data operations for DPLA ingests, ensuring correct profiles and up-to-date hub data.

Core Features & Use Cases

  • Find latest data in S3 and determine fresh harvest/mapping/jsonl exports for each hub.
  • Sync hub data to S3 using the project scripts to ensure correct prefixes and anomaly detection.
  • Check JSONL exports sync status and verify consistency between local exports and S3.

Quick Start

Sync the latest hub data to S3 using the dpla profile.

Frequently Asked Questions about dpla-s3-ops

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I sync hub data to S3 for the DPLA ingestion workflow?▼

Sync hub data to S3 by executing the dedicated scripts under scripts/ using the AWS profile dpla. Sourcing the repo environment ensures correct prefixes and applies anomaly detection for deterministic results.

What is the best way to verify JSONL exports consistency between local files and S3?▼

Verify JSONL exports consistency by running the specific scripts to check the sync status between local exports and S3. This ensures data integrity by comparing local JSONL files against the latest S3 snapshots.

How do I find the latest data snapshot in S3 for each hub?▼

Find the latest data snapshot in S3 by running the retrieval scripts to determine fresh harvest, mapping, and JSONL exports for each hub. This requires sourcing the repo environment and using the dpla AWS profile.

Do I need a specific AWS profile to manage S3 data operations for DPLA?▼

Yes, using the AWS profile dpla is required for managing S3 data operations for DPLA. Sourcing the repo environment alongside executing dedicated scripts ensures deterministic results and correct data synchronization.

Why does my S3 data synchronization return inconsistent results?▼

S3 data synchronization returns inconsistent results if the repo environment is not sourced or if dedicated scripts are bypassed. Executing the correct scripts under scripts/ with the dpla profile ensures correct prefixes and anomaly detection.

When do I need to sync hub data to S3 using project scripts?▼

Sync hub data to S3 using project scripts when you need to automate S3 data operations for DPLA ingests. This process ensures correct profiles, up-to-date hub data, and verifies JSONL export consistency.

Related Skills

Explore More Data & Analytics Skills→
dpladpla
35

dpla-s3-and-aws

Coordinate S3 data synchronization and AWS operations for the DPLA ingestion pipeline.

Official
Intermediate
dpladpla
35

s3-latest

Identify and retrieve the most recent S3 exports for a specified hub.

Official
Basic
dpladpla
35

dpla-ingest-debug

Diagnose and fix DPLA hub ingestion failures across pipeline stages.

Official
Advanced
winsorllcwinsorllc

s3-tools

Manage files and buckets in S3-compatible storage using AWS CLI.

Community
Intermediate