managing-imports

File staging automation organizes incoming files by type and term, then checks file hashes against a PostgreSQL database to prevent duplicate imports. This skill handles the complete workflow from inbox classification through archiving, using SHA256 hashing and the file_staging orchestrator to track status and maintain data integrity.

Duplicate detection compares incoming file hashes against an import_manifest stored in PostgreSQL, blocking already-processed files before they enter the pipeline. This approach eliminates manual verification and ensures each import runs only once.

Import status tracking provides a clear overview of files at each workflow stage—inbox, staged, processing, and archived. The skill generates real-time reports showing queue depth and processing state, reducing manual oversight in your data pipeline.

Yes, PostgreSQL stores the import_manifest for duplicate detection, and psycopg2 connects Python to the database. You also need local inbox, staged, and archive directories and python-dotenv for environment configuration.

Term detection classifies incoming files by academic term during staging, organizing them into appropriately named directories. This enables term-based sorting and archive structure, supporting multi-term data pipelines without manual categorization.

Processed files are automatically archived to an organized structure keyed by term, removing them from active staging directories. This keeps your inbox and staged paths clean while preserving a complete audit trail for historical reference.